Actions
Engagement
  • Author Author: colporteur
  • Date Created: Date Created
  • Last Updated Last Updated:
  • Views 1597 views
  • Likes 0 likes
  • Comments 20 comments
Related
Recommended

When was the last time you used sed and/or awk?

I'm thinking of preparing a lecture, for a High School Computer Club, on the command line tools sed and awk for data manipulation. The discussion will include some examples of using the command to mung a data set. I found the commands valuable over my technology career but wonder how relevant are they today?

  • in reply to jomoenginer

    The real unix guys at work would call me a micky mouse because I used online interactive regex test pages to test my expression, instead of terminal and keyboard image.

  • in reply to shabaz

    Interesting fact from the 'AWK' Wiki site, AWK was created at Bell Labs in the 1970s,[6][better source needed] and its name is derived from the surnames of its authors: Alfred Aho, Peter Weinberger, and Brian Kernighan.

     

    I believe there were a couple more commands which were derived from initials of the developers.

  • in reply to neilk

    And rarely do they have non-cryptic names.

    There's something for everyone in Linux, worth playing around with (say) a Pi's command line to explore, but finding all the interesting commands can take a lifetime. Just discovered the other day for the first time a command to search for human-readable text among the noise.. called strings

    There's the obscure-sounding troff and groff, useful for converting text into printer format. And myriad other weird commands. nm is very cool, I used to use it a lot.. it searches binary files and tells you the names of all the functions inside.

    One command I wanted, but never found (I'm sure it could be fashioned from sed/awk very easily from those in the know, but I wouldn't know how!) was a command to hunt rows of data that definitely contained text from a 'good list' but to definitely not include the lines from a 'bad list'. The reason being, it would be useful for quickly going through large log files, by first setting up the good and bad lists. The good list could contain (say) "ERROR" and "DEBUG" and the bad list could contain "LEVEL1" and then you'd see only errors or debug lines that were level 2 or higher (for instance). Since it was too hard to figure out with regexp and Linux tools, I did it in C++ using the normal string library.

  • Sadly, I've never come across either!

     

    Neil

  • It must be 20 years since I used AWK.

     

    I still have the book.

     

    MK

  • in reply to shabaz

    The flexibility of command line tools...

    and cascading command line functions with the pipe command makes it even better.

  • in reply to genebren

    The flexibility of command line tools with all the switches is always great to see : ) Apparently, some hacker was once caught because they happened to use an unusual combination of switches on 'ls' on the system they compromised.. (e.g. always typing "ls -al" instead of "ls -la" and so on).

    I like vi too : ) use it several times a week.

  • AWK is one of my go to tools for processing text.  I have used it in the past for a variety of projects, including some data analysis projects.  In one case, I used it to reduce large data sets (engine and airframe models) by finding the minimal number of data points necessary to interpolate (multiple methods - linear, bilinear, cubic and circular) data within a given error specification.  This reduced the development time over prior method of curve fitting and produced greater accuracy and faster execution time in the flight management controller systems that we were designing.

     

    I have a couple of tool sets that allow me to use these tools on my windows-based computer and I use them somewhat frequently.  Having spent a lot of time developing software on UNIX/LINUX, I have fond memories using a variety of these tools. with my favorites being VI, GREP, AWK and FIND.

  • Had you asked a few weeks ago, I would have said I last used it about 10 years ago. But someone asked me a sed question a few weeks back. I think it's handy having knowledge of such tools, and the underlying regexp is still handy to know too, but today I cannot recall much except say how to identify the beginning of a string, and how to match on a set of characters, etc. But I still have a regexp book somewhere just in case the need crops up.

    Since you mention specifically data manipulation, if a programming language is preferred then there are preferred or popular languages. For that, Python would be among the top few preferences for data manipulation for many I think (and there are more specialized languages too, I'm just picking an all-rounder) - and is encountered in many places, not just on a computer's command line. So today for me that would be my preferred choice if I had a choice, but I wouldn't feel extremely uncomfortable if I had to do one-off manipulation with sed occasionally. Personally I wouldn't use it for running (say) regular jobs, I would prefer a program in say Python, but that's just me, others would have other preferences.