software engineering;  was: Is this official blog article intended as a joke?

Actually, the 'blog makes a useful point:

Most computer science students get to learn high-level programming languages and application development but they don’t really understand computers.

JMO/YMMV:

Now, while interfacing to the real world is fun, RasPi doesn't really address the high-level application problem.  It's still a GNU/Linux box, running a mainframe operating system with dozens of processes.  It's darn cheap for a mainframe, but it's still a mainframe.

If you really want to learn what computers do IMO you really need to learn assembly language and run it on a real mini- or micro-computer with no operating system present to protect you from stomping all over your code and data so that your program goes off to la-la land.  That's when you really see that all that's in computer memory is bit-strings, and the computer will interpret them as instructions or data depending on where you told it to execute -- whether you meant to or not.  Once you've mastered the machine-language level, then you can understand why your C program is misbehaving in that extraordinary fashion.  IMO this mastery is necessary for someone to be called a computer scientist, just as you expect an EE to know how to use Kirchhoff's Laws and a mathematician to be able to do Epsilon-Delta proofs.

> IMO this mastery is necessary for someone to be called a computer scientist,

Of course, computer science is a big field, with many specialties, and many layers

of abstraction, and it's entirely possible for hardware specialists to know very little

about software, and vice versa.  And theoretical computer scientists often know very

little about either software or hardware, focusing their attention on topics like

algorithmic efficiencies that don't provide any benefit on realisitic sized data sets.

Dijkstra famously said "computer science is no more about computers than

astronomy is about telescopes."

Of course, it's still important to know how to read an abend dump when you have to.

The solution to complexity is abstraction.  A single person can run a large company

or run a large nation, but not by doing all the jobs that a sole proprietor does.  Instead,

they operate at much higher levels of abstraction.

I don't think the analogy of "sharp tools" to "low level programming languages" works too well.

Certainly low-level features get removed from higher level languages, for example, gotos have

been removed from recent languages.  And if you focus on the loss, it may seem like your language

has gotten duller.  But that's usually not true, because new higher-level abstractions are added

as lower-level ones are removed.

In some cases, the benefits are not obvious to the programmer, but they are to the compiler

writer, because keeping lower-level abstractions around allows them to be used to subvert

the higher-level abstractions.

For example, the typical compiler intermediate language these days is based on SSA,

and that requires a control-flow graph in the form of a DAG (no cycles), which in turn requires

that you don't have arbitrary gotos, such as between branches of an if/then/else construct.

So by taking away the low-level goto, and replacing it with higher-level if/then/else and looping

constructs, you end up with a language that is no less capable, but much more suitable for

SSA-based optimization, code generation, and other analysis.

coder27 wrote:

 

The solution to complexity is abstraction.  A single person can run a large company

or run a large nation, but not by doing all the jobs that a sole proprietor does.  Instead,

they operate at much higher levels of abstraction.

Abstraction is part of the solution, but it's not sufficient by itself.  With abstraction alone, computers will continue to do exactly what we tell them to do, and will inevitably end up doing the wrong thing because it is entirely beyond our mental ability to fully understand (let alone specify) the behavior of a non-trivial system.  Removing unnecessary detail and bringing the language of specification closer to the natural language of the application domain through DSL abstraction clearly helps, but it is not enough.  The desired complex behavior still has to be defined, somewhere, and that's the problem.

Your example of the large company or nation is a very good one, but not as an illustration of the power of abstraction.  What it illustrates instead is that complex systems can work well when there is semantic commonsense built into the components of the system, so that daft commands from above don't cause an immediate collapse.  And that's what is lacking almost completely in software today.

Remember Bertrand Meyer's very insightful work on Eiffel, in which preconditions, postconditions, and invariants played such an important part?   He embedded semantic constraints into software components, although his focus was on simple design contracts at a level far below anything related to what people might call "commonsense".  Unfortunately, static constraints are fundamentally at odds with flexibility and reuse, so we see them only very rarely in practical software beyond basic assertions for defensive programming.  We are as far away as ever from the model of companies or nations working in software.

John Beetem wrote:

 

Actually, IMO humans are quite good at working with complex things once they have come up with good notations -- textual and graphical -- to describe them.  However, the development of those notations usually takes decades or centuries.

It's interesting (or perhaps sad) to contrast that with the typically short  timescales of product and company planning in the commercial software world.  One might be tempted to suggest that open source could be the answer, since it doesn't end-of-life code with quite the same degree of profit-oriented ruthlessness, but alas even open source programs bit rot and stop working owing to dependencies and changing architectures.

The long-term view just isn't there, and may not even be possible.  Things change, it's part of progress.  Whatever solutions we come up with for the software crisis, they'll need to be able to cope with the current state of technology.  I very much doubt that refinement of languages or written code over ages will help much.

Morgaine Dinova wrote:

 

coder27 wrote:

 

The solution to complexity is abstraction.  A single person can run a large company

or run a large nation, but not by doing all the jobs that a sole proprietor does.  Instead,

they operate at much higher levels of abstraction.

 

Abstraction is part of the solution, but it's not sufficient by itself.  With abstraction alone, computers will continue to do exactly what we tell them to do, and will inevitably end up doing the wrong thing because it is entirely beyond our mental ability to fully understand (let alone specify) the behavior of a non-trivial system.  Removing unnecessary detail and bringing the language of specification closer to the natural language of the application domain through DSL abstraction clearly helps, but it is not enough.  The desired complex behavior still has to be defined, somewhere, and that's the problem.

Another problem is that of leaky abstraction layers.  My favorite example of this is C.  As I said earlier in this discussion, I think of C as a portable assembly language which uses high-level notations, and if you think of C that way it works very sharply indeed.  However, if you think of C as a high-level language, you will be disappointed at run time when machine-level behavior damages your nice high-level model.

Just adding abstraction layers can be problematic, if only because the abstraction layer may hide terrible performance problems that you'll only uncover when the system is fully deployed.

Also, IMO opinion many large (and even small) companies and countries are very poorly run whether or not their leaders are capable of abstract thinking or common sense.  I'm sure you can think of examples.  I believe the principal cause of this is the Law of Large Organizations:

In any Large Organization, Loyalty will always be rewarded over Competence.

Come to think of it, many large software systems suffer from this as well

Morgaine Dinova wrote:

 

John Beetem wrote:

 

Actually, IMO humans are quite good at working with complex things once they have come up with good notations -- textual and graphical -- to describe them.  However, the development of those notations usually takes decades or centuries.

 

It's interesting (or perhaps sad) to contrast that with the typically short  timescales of product and company planning in the commercial software world.  One might be tempted to suggest that open source could be the answer, since it doesn't end-of-life code with quite the same degree of profit-oriented ruthlessness, but alas even open source programs bit rot and stop working owing to dependencies and changing architectures.

 

The long-term view just isn't there, and may not even be possible.  Things change, it's part of progress.  Whatever solutions we come up with for the software crisis, they'll need to be able to cope with the current state of technology.  I very much doubt that refinement over ages will help much.

Open source certainly helps, since it's not subjected to the limitations imposed by having to make money

Ars longa, vita brevis.

John Beetem wrote:

 

Open source certainly helps, since it's not subjected to the limitations imposed by having to make money

Unfortunately open source + free software is not fully isolated from the ill effects of preoccupation with money, short-term "product" focus, business planning and corporate competition.  Although individual projects may be created and developed by one or a small number of non-commercial and wholly altruistic enthusiasts, as soon as a project blooms into a global community resource it gains contributors with different needs and aspirations, many of them in direct support of business requirements and a focus on product and competitive advantage.

Open source code inevitably gets modified to support the latest devices or architectures, which typically undoes any previous validation that may have been done on it.  New drivers or patches are often added by corporate developers for whom getting support for a new gizmo is part of their jobs, but who will generally not continue to support their changes once those gizmos have been EOL'd in the company product cycle since they will not be paid to do so.  And unless they are exceptionally community-minded, they'll not be going out of their way to create generic infrastructure that supports their competitors, but instead focus on providing best support for their own requirements as their main priority.

So, open source doesn't exist entirely within an isolated collaborative space as is sometimes portrayed.  Even with everyone doing their level best and not having hidden agendas nor being deliberately evil, the nasty realities of the world outside FOSS idealism do intrude and mess things up to some degree.  Old code that is perceived as crufty is often removed purely to keep the more modern code sane, which means that honing code to a high state of reliability and trustworthiness over long periods of time just doesn't happen --- it bit rots or gets modified long before then, so validation and long-term stability just isn't useful or even possible in practice.

This is why I say that any solution to the software crisis cannot come from the "decades or centuries" that you mentioned, not even in FOSS.  It will have to be applicable to the code that exists at any given point in time or it won't be addressing the problem.

Morgaine.

Morgaine Dinova wrote:

 

John Beetem wrote:

 

Open source certainly helps, since it's not subjected to the limitations imposed by having to make money

 

Unfortunately open source + free software is not fully isolated from the ill effects of preoccupation with money, short-term "product" focus, business planning and corporate competition.  Although individual projects may be created and developed by one or a small number of non-commercial and wholly altruistic enthusiasts, as soon as a project blooms into a global community resource it gains contributors with different needs and aspirations, many of them in direct support of business requirements and a focus on product and competitive advantage...

Good points.  However, I would say that FLOSS (Free-as-in-Liberty Open-Source Software) generally benefits from participation by for-profit entities as long as the software uses a copy-left license.  Yes, money-driven entities will want to add code for their own purposes, but if they want to use immature FLOSS for serious purposes they'll find bugs and they'll be required by the license to publish the fixes to those bugs.  The rest of the FLOSS community can scoop up the bug fixes and improve the quality of the software.  It's very hard for an individual developer or even a small team to catch and fix everything, so having people paid to help is usually beneficial.

Example number 1 is obviously Linux.  It has grown amazingly over its lifetime, both in use and... in size.  I think of Linux as a terrific mainframe OS kernel, and when you think of it that way then its size isn't so shocking.  I think there's room for a light-weight personal computer OS, but I think we'll have to wait for Microsoft to complete its self-destruction first.  I don't think the wait is going to be long -- they seem pretty eager.

Linux benefits from having a foul-mouthed benevolent dictator to enforce quality, and from early adoption by IBM who defended Linux from early attempts to destroy it.

Example number 2 is Open Office.  That's a good example of a project that had to "fork it" to remain FLOSS.  I agree with RMS that you can't get the benefits of open-source software without the Four Freedoms, because otherwise corporations will take a snapshot of OSS and then keep all improvements to themselves.  It will be interesting to see how long Open Office stays around versus Libre Office.

Remember Bertrand Meyer's very insightful work on Eiffel, in which preconditions, postconditions, and invariants played such an important part?   He embedded semantic constraints into software components, although his focus was on simple design contracts at a level far below anything related to what people might call "commonsense".  Unfortunately, static constraints are fundamentally at odds with flexibility and reuse, so we see them only very rarely in practical software beyond basic assertions for defensive programming.  We are as far away as ever from the model of companies or nations working in software.

Yes, I'm somewhat familiar with Eiffel's semantic constraints, but I've never heard that "static constraints

are fundamentally at odds with flexibility and reuse", so I'd be interested in further elaboration or references.

I am aware of various issues such as whether constraints should be statically checked at compile time vs.

run-time checked.  It seems both are needed.  I'm also aware of some confusion when inheriting constraints,

whether you should inherit the union of the constraints of all your ancestors, and how to override any constraints

you don't want to inherit.  But I assume that constraints are relatively rarely used in practice due to their availability

in mostly non-mainstream languages.

In any case, I think abstraction as a solution for complexity doesn't necessarily depend on the power of

available constraint mechanisms.  It mostly depends on the suitability of divide and conquer, which allows

for a hierarchical decomposition of the problem.  A CEO can run a company because the company can be

divided into various departments, each with a vice president, and with limited interaction between departments.

coder27 wrote:

 

In any case, I think abstraction as a solution for complexity doesn't necessarily depend on the power of

available constraint mechanisms.  It mostly depends on the suitability of divide and conquer, which allows

for a hierarchical decomposition of the problem.  A CEO can run a company because the company can be

divided into various departments, each with a vice president, and with limited interaction between departments.

In 1984, there was a USA TV comedy series called Empire about boardroom shenanigans at an unnamed USA corporation.  The hero is a young man from R&D who was made VP because of competence, and you get to watch him try to swim among the sharks.  It only ran 6 episodes -- I think it struck a nerve with the sponsors.  Not yet on DVD, alas.

coder27 wrote:

 

Yes, I'm somewhat familiar with Eiffel's semantic constraints, but I've never heard that "static constraints are fundamentally at odds with flexibility and reuse", so I'd be interested in further elaboration or references.

Oh it's nothing particularly deep, just an inherent consequence of adding constraints to any code.

In functions for example, without additional constraints, the domain of a particular function is given by the types of its declared parameters (or the types of its supplied arguments if function parameters in the language are untyped), and so those input types determine the extent of the function's input space.  The range of the function (ie. its set of possible outputs or the extent of its output space) then follows naturally from the set of possible inputs constrained only by the definition of those input types.

If you add additional constraints within the function to narrow down the allowable function domain to less than the full input space given by its input types then the output range will frequently be more limited as well, ie. the extent of the output space will be smaller.  Typically this is done to bring the semantics of the function more into line with the desired behavior of the application, for example to narrow its domain and range for the purpose of defensive programming.  The function will then have a more limited set of possible uses than would its original version that is constrained only by its input types, which is why adding semantic constraints is fundamentally at odds with flexibility and reuse.

coder27 wrote:

 

Yes, I'm somewhat familiar with Eiffel's semantic constraints, but I've never heard that "static constraints are fundamentally at odds with flexibility and reuse", so I'd be interested in further elaboration or references.

Oh it's nothing particularly deep, just an inherent consequence of adding constraints to any code.

In functions for example, without additional constraints, the domain of a particular function is given by the types of its declared parameters (or the types of its supplied arguments if function parameters in the language are untyped), and so those input types determine the extent of the function's input space.  The range of the function (ie. its set of possible outputs or the extent of its output space) then follows naturally from the set of possible inputs constrained only by the definition of those input types.

If you add additional constraints within the function to narrow down the allowable function domain to less than the full input space given by its input types then the output range will frequently be more limited as well, ie. the extent of the output space will be smaller.  Typically this is done to bring the semantics of the function more into line with the desired behavior of the application, for example to narrow its domain and range for the purpose of defensive programming.  The function will then have a more limited set of possible uses than would its original version that is constrained only by its input types, which is why adding semantic constraints is fundamentally at odds with flexibility and reuse.

The function will then have a more limited set of possible uses than would its original version that is constrained only by its input types ...

OK, I understand.  I think normally this is no great loss, because the region of the domain

that is trimmed is usually erroneous or uninteresting.  For example, adding a constraint to ensure

that a parameter is nonzero, because you intend to use it as a divisor.  This allows pushing the

divide-by-zero check to the caller rather than the callee, which is a good thing, because the

compiler may be able to remove it based on information known at the call site.