February 2004 (#99):

or read TWDT


The Mailbag


HELP WANTED : Article Ideas
Submit comments about articles, or articles themselves (after reading our guidelines) to The Editors of Linux Gazette, and technical answers and tips about Linux to The Answer Gang.


Bash Scripting

Sat, 07 Jun 2003 20:01:53 +0530
Robin Chhetri (robinchhetri from fastmail.fm)

Hi Gang, I asked another question around one month ago and even though I could not come up with a answer I decided to come up here again.(Incidentally I tried it also in linuxquestions.org). What would be the equivalent to the zsh script given below in bash

$echo ${${(z) $(whereis libcrypto)}[2]}

if the output of $(whereis libcrypto) is

libcrypto: libcrypto.so libcryto.a
it returns libcrypto.so only.

Now I could come up with

$robin=($(whereis libcrypto)); echo ${robin[1]}

But can it be done in one go using some construct? I am not a shell guru so I wonder if it can be done!

Robin


Article suggestion - Clustering (formatting correction)

Wed, 31 Dec 2003 04:05:55 -0800
Dave Bechtel (kingneutron from yahoo.com)
Answered By Heather Stern, Ashwin

Dear God, yahoo really b0rked that one. All because I wanted to underline .net... Sorry 'bout that.


--I've been vaguely interested in clustering for a while, but really don't know what I could do with it. I have 3 machines:

--I can't recompile a kernel intended for the slow Intel machines on the fast AMD, it just doesn't work. However, I could go out and buy some cheap machines at a computer show for ~$30-$40 apiece, or get some loaners from a friend.

--My question is, what are clusters good for besides graphics / animation / video editing / number crunching (none of which I'm interested in)? Can I gzip/ bzip2 compress a 4-8GB tar file on a distributed cluster and save time? Can I recompile a 2.4 kernel that way so it doesn't take an hour and a half? (Yes, even using "make -s -j 3".)

--I've heard of OpenMosix, but haven't looked into it very far. Just a note, my network is 100MBit Ethernet. Any info the LG.net folks could provide would be welcome, TIA.

===== Contents above ThisLine (C)ThisYear KingNeutron Ltd.

[Heather] We'll need permission to publish your message and responses related to the thread for the world wide webzine Linux Gazette, it you want a decent chance at an answer.
This is clear enough to post as a Wanted if none of the Answer Gang want to take a pop at it, but we're past deadline for the current issue; it'd end up in February.
And of course, he did grant that, and here you have it. But it's worth noting to folks who copyright their usenet posts, we need your intent to be public or we can't publish it :) And if we can't publish it, we often won't answer it either.

===== Check out KNOPPIX Debian/Linux 700MB Live CD: ===== http://www.knopper.net/knoppix/index-old-en.html "C00K13 M0N573R 0WNZ J00!! PH34R C00K13 M0N573R 4ND 0SC4R 4ND 3LM0 4ND 5NUFFL3UP46U5 4ND 7H3 31337 535AM3 57R337 CR3W!!" .dotgoeshere.

[Heather] To the tiniest portion of an answer: there's a mosix aware variant of Knoppix. Check out LWN's Distributions page. (lwn.net/Distributions) If you get things going, I agree wholeheartedly -- this could make a really fun article by you about getting your cluster going. Are you suggesting that you could write this, or that you would like to see the topic come up?
As such I've changed the gazette@ (main editor) cc to articles@ (article ideas and submissions)
[Ashwin] Have you tried Cluster Knoppix?
http://bofh.be/clusterknoppix


RV: Sendmail Help

Sun, 25 Jan 2004 17:52:44 -0500
Juan Carlos Diez (jdiez from eluniversal.com)
Answered By Thomas Adam, Mike Orr (Sluggo)

Dear Ben, hello.

My name is Juan Carlos Diez, a novice Unix Sysadmin with no experience at all with sendmail, who needs desperately your kindly help :-)

I read your sendmail notes on http://www.linuxgazette.com/issue58/okopnik2.html and I thought maybe you could help me.

Currently we have a Red Hat v6 server with sendmail v 8.9.3 running. We have set a new server with Red Hat 7.3 and sendmail 8.11.6 in order to migrate all of our services from the current server to the new one.

My question is: May I use the current sendmail.cf and sendmail.mc files in the new server to avoid manually configuring sendmail again? I mean, to copy such files to the new server and restart sendmail, do you think it will work fine?

Thank you very much.

[Thomas] Why have you not tested it yourselves? You certainly have nothing to loose by doing so :) To answer your question though, you will have no problems using the configuration files. They're based in a rather interesting scripting language called m4 which is a separate entity to sendmail.
[Sluggo] He may not know what TAG is, Thomas. :)
TAG is The Answer Gang, the group of volunteers at Linux Gazette that answers tech-support questions. All questions and answers are considered for publication in a future issue. Please address follow-ups or future questions to tag@linuxgazette.net. More information about The Answer Gang is at: http://linuxgazette.net/tag/ask-the-gang.html

Thank you all for answering, I really appreciate it.

Best regards,

JCD.


GENERAL MAIL


Jython article

Thu, 13 Nov 2003 22:09:46 -0500
Rob Tougher (robt from robtougher.com)

BTW, thanks for your comments on the article. Your editorial
efforts make LG a better magazine.

Editorial oversight does matter. People will see that in the magazine's quality.

- Rob


Windows Defectors

Mon, 5 Jan 2004 16:32:06 +0000 (GMT)
Tony Dearson (ajdearson from lycos.co.uk)

I would disagree with a softening of the advise never to log in as root. Respectively, I would remind that there are two types of Linux/Unix Administrators --

1. Those who have trashed the entire system by mistake.

2. Those who will trash the entire system some time in the future.

Personally, I avoid logging in as root as much as possible, preferring to su for a brief perioed when necessary.

Kind regards
Tony Dearson

[Sluggo] I don't see much difference between logging in as root vs su'ing. Root is root.
[Ben] Ever dip your finger in liquid nitrogen, Mike? It's a geek thing, FSVO "geek" (e.g., radars and black-body targets.) In-and-out, no problem. Keep it there for a full second or so, and you'll have frostbite that may require amputation. (Five seconds or so, and you'll be able to shatter it like glass.) When you log in as root, everything you do - and the chain of consequences that proceeds from it - is done as root, and you must consider that consequence tree for every single command you type including "ls". When I type a command, then realize that I need root privs for it, I just hit the up-arrow, type:
"<Ctrl-A>su -c "
think about the effects, and press 'Enter'. Somehow, typing 10 characters (7 if I know it beforehand) does not seem to me to be the Sysyphus' burden (with a bit of Prometheus thrown in for effect) that you insist it is.
[Sluggo] I hate typing "su[Enter][password][Enter][command][Enter][ctrl-D]" all the time, or even worse, "su -c 'command in quotes'[Enter][password][Enter]".
[Ben] [shrug] Everyone has their pet hates, of course. This does not in any way correlate to what makes for reasonable system usage.
[Sluggo] I love the fact that konsole has a "Root console" menu option, so I can get a root session any time without using the arcane su syntax.
[Ben] In what way is it arcane? Is "-c" in some way connected to the Norse Edda and the Boghaz-keuy Babylonian tablets? I wasn't aware. Molehill -> mountain requires a sufficient amount of dirt, and I'm afraid there just isn't enough.
[Sluggo] (You do have to type the root password, of course.) Even with that, I usually leave the session open in the background for a while in case I need it again.
The most important thing I ever learned (from sysadmin Pann McCuaig if he's reading) is, sit on your hands before pressing [Enter] after any potentially destructive command like "rm -rf", "rsync", etc.
[Ben] [grin] I prefer to use my other end to control my hands when using root privilege, thanks.
[Sluggo] Make sure you're the right user, on the right system, and in the right current directory. (Actually sitting on your hands is optional, but the metaphor is good so you don't forget it.)
If all that information doesn't appear in your shell prompt, read the manpage for your shell and set PS1 or PROMPT accordingly. My favorite prompt strings for zsh are:
PROMPT="%S%n@$HOST_:%~%#%s "
PROMPT2='%S%_>%s '
For root I use bash with these lines:
export PS1='ROOT@`hostname`:`pwd`$ '
export PS2='> '
[Ben] I've done the following in my "~/.bashrc":

See attached ben.bashrc.txt

Whenever I'm root, my text is all in red. Makes for a decent clue, hard to miss.
[Sluggo] (Normally people use '#' to distinguish their root prompt, but I find that too easy to miss. I also need '$' to show it's bash rather than zsh (whose conventional symbol is '%').
Another important thing to remember is that "su -" gives you root's (or anybody's) full shell environment,
[Thomas] ...so that $SHELL, and other ~/.profile files, etc are sourced, as well as various other exported $VARIABLES are updated.
[Sluggo] while "su" alone gives you a partial environment that doesn't include all their environment vars. E.g., "echo $USER" (or "echo $LOGNAME" in some shells) shows your login rather than root's. That may cause some programs to do the Wrong Thing; e.g., 'mutt' will read your mail rather than root's.
[Thomas] Will not change $SHELL, and other variables.
This can be negated with the:
su -m
switch.
[Sluggo] (No, you still shouldn't read mail as root anyway! Put those dark classes back in your pocket, Ben.)

Mike,

That's a good tip, thanks. A small detail I've never come across before

Tom


About solving priblem of /bin/bash access denied

Wed, 7 Jan 2004 20:56:46 -0500
sameer sonaikar (sonaikar from yahoo.com)
Privately sent to Ben, and forwarded to the Linux Gazette for publication. -- Heather

Dear sir,

I am using Red Hat 7.2 and facing same problem after installing sendmail. Might I have made mistake while confugering Sendmail. But I am trying to find the mistake.

I got a lot Encouragement from ur efforts.

Thanking you.

With Regards,
Sam

[Thomas] Are you saying that you followed Ben's article (http://linuxgazette.net/issue52/okopnik.html) and that after you installed sendmail the problem arose that /bin/bash access denied? That being the case I would either do:
rpm -qilp sendmail
to generate a list of files belonging to that package, and check the perms for each (assuming you're using an RPM-based distro), or under debian:
dpkg -L sendmail
I am deliberately being vague here.
Your question as it stands lacks structure, information and meaning. I suggest you look here:
http://linuxgazette.net/tag/ask-the-gang.html
for further information.
-- Thomas Adam

GAZETTE MATTERS


Clear writing

Thu, 1 Jan 2004 21:29:08 -0500
Ben Okopnik (the LG Answer Gang)

I'm not sure where this should be shoehorned into our FAQs, but - having just read this excellent 25-page paper, I think it should be required reading for prospective authors (or anyone who wants to write a technical paper). "Clarity in Technical Reporting", which had a long run as an underground publication at NASA, was officially published by The Powers That Be once they caught on (thus proving that they were worthy of their positions. All hail, etc.)

http://techreports.larc.nasa.gov/ltrs/PDF/NASA-64-sp7010.pdf


stylesheet problem??

Fri, 02 Jan 2004 10:39:15 -0700
Bob van der Poel (bvdpoel from kootenay.com)

Hi all. Just grabbed the #97 and am having some problems reading it on Mozilla 1.6b on Mandrake 9.0. The navigation box overlays the first several lines of each article. I played a bit with lg.css (which I don't know anything about) and got it readable by outcommenting the line:

	position: absolute;

at line 53.

Now, the article text overlays the nav. box. Not what you intended, but it is readable... a real fix would be appreciated. Hey, this bug might even spur me on to learn how to use stylesheets :)

And: keep up the great work!

[Mike] Thanks. Our stylesheet editor, Rob Tougher, is away on family business, and I have only a rudimentary knowledge of CSS. Are the menu links showing up one per line with a black bullet left of each? And the breadcrumbs (=Yahoo bar) too? That was something I specifically fixed yesterday, because our stylesheet fixes this month (for Netscape 4 and Phoenix) broke compatibility with issue 97. If you do "View Source" on an offending page and scroll to the bottom, the menu links should not have <li>...</li> around them. If they do, your browser is caching an obsolete version of the page.
I tried your solution but it breaks in my Galeon: it moves the menubar to the very top, covering part of the logo and Tux. While this is readable, it's not acceptable. Thanks for letting us know anyway, and for any other ideas you might have later.
A new problem we have is that after I added the "Contact Us" link, the menubar is extending too far to the right past Tux. I couldn't figure out how to prevent that. Do you have any ideas?

Well, problem is solved. I dl'd the version on the site just now and it works just fine. Guess I grabbed while you were putting, or something. Thanks!


Linux Gazette

Sat, 3 Jan 2004 10:16:33 +0000
Martin J Hooper (martinjh_linux from blueyonder.co.uk)

Heather,

Been reading for a while and lost the site when you started having problems. Nice to find it again!

Keep up the good work. :)

Martin

Thanks, Martin, I'm glad you like how we've kept it. Happy New Year :) -- Heather


how to join the TAG mailing list?

Wed, 31 Dec 2003 17:29:51 -0800
sanjaya singharage (sanjayas from opensource.lk)
Answered By Mike Orr (Sluggo), Breen Mullins, Ben Okopnik, Thomas Adam, Heather Stern

This is a multi-part message in MIME format.

You'd think that if Sanjaya had been on this list before that he would have remembered that none of us appreciate MIME encoded e-mails.... oh well...
I guess a beneficial reading of:
http://linuxgazette.net/tag/ask-the-gang.html
is in order. -- Thomas
I used to be on the tag mailing list some time ago. Does it still exist? How can I join it? thanks.
[Mike] http://linuxgazette.net/mailman/listinfo/tag
We couldn't carry the subscribers over from the old linux-questions-only list because SSC wouldn't give us the member roster. Now the list is fully under our control.
[Breen] By the way, I'm back too. It's been pretty busy IRL, but I finally got around to moving here.
Cheers and Happy New Year,
doesn't seem obvious to find http://linuxgazette.net/mailman/listinfo/tag on the home page of http://linuxgazette.net Did I miss something?
Wishing all a happy new year!
sanjaya.
[Ben] It's not supposed to be obvious; you were just shown a shortcut. If you want to join, the what you're really supposed to do is read the TAG FAQ at:
http://linuxgazette.net/tag/members-faq.html
which will direct you to the subscription address, etc. after explaining the necessary facts of life - such as, you do not need to subscribe to TAG in order to ask a question. Subscriptions to TAG are for those who are willing and able to answer Linux questions.
Good point - we should update the "About TAG" blurb on the mailman link given above to include this concept, give the hotlink to member policy, and advise querents to read ask-the-gang and then just send mail. -- Heather


bum link

Thu, 15 Jan 2004 10:01:22 -0800
Carla Schroder (carla from bratgrrl.com)

Thanks. We've been debating how much to change the back issues. Plus it's a big job, and we can't change the mirrors that don't resync back issues.

Hello Mike,

No worries. :) Just letting you know, what you do about it is up to you.

"The Wonderful World of Linux 2.6" is absolutely incredible. What a great article.

cheers


This page edited and maintained by the Editors of Linux Gazette
HTML script maintained by Heather Stern of Starshine Technical Services, http://www.starshine.org/

Published in Issue 99 of Linux Gazette, February 2004


More Two Cent Tips!

See also: The Answer Gang's Knowledge Base and the LG Search Engine


no-floppy system configuration

Thu, 8 Jan 2004 11:47:45 -0500
Ben Okopnik (The LG Answer Gang)

Many new laptops come without a floppy drive; the one that I just bought, an Acer Aspire 2003LMi, does not have one - it's available as an extra option, but I don't see myself needing it. However, part of my standard method for converting Win-machines to dual-boot involves using FIPS, which I use to "shrink" the Wind0ws partition to a minimal size - and FIPS normally runs from a floppy. What to do?

Here's an interesting fact that many people may not be aware of: the bootable part of a CD consists of nothing more than a bootable floppy image. So, I simply took a DOS boot floppy containing FIPS, and made a byte-by-byte copy:

# Create a directory to hold the CD data (none at the moment) plus "boot"
ben@Fenrir:~$ mkdir -p /tmp/cdrom/boot
# Create byte-by-byte copy in "boot.img"
ben@Fenrir:~$ dd if=/dev/fd0 of=/tmp/cdrom/boot/boot.img

I then created an ISO image containing that disk copy. If I wanted any other data on that CD (I'll probably make another one with a bunch of DOS utilities on it later; I've been using bootable DOS "tool" floppies for over 20 years to repair broken Wind0ws systems), I'd copy that data into "/tmp/cdrom", and it would become part of that image.

ben@Fenrir:~$ cd /tmp/cdrom
ben@Fenrir:/tmp/cdrom$ mkisofs -r -b boot/boot.img -c boot/boot.catalog -o bootcd.iso .

All that was left was to burn the newly-created image to a CD:

ben@Fenrir:/tmp/cdrom$ sudo cdrecord -v -eject speed=8 dev=0,0,0 bootcd.iso

The only downside to this is not being able to save the boot sector to the floppy before repartitioning... but in the worst case, it's a new system without any of my data on it, and it's not a concern. Besides, I have Linux, and boot sector recovery is rather trivial. :)


Nice value in XFree86 startup

Sat, 24 Jan 2004 21:56:43 +0000
Thomas Adam (The LG Weekend Mechanic)
Question by Jacobo ;-) (jacobo221 from hotmail.com)

Hi, first of all, sorry for my english, I try my best, but I use to fail in spelling and grammar ;-) Well, I'm writting to you because I have a problem at Xfree86's startup. I'm useing kernel 2.4, so I have, in Xwrapper.config, nice value set to -10 ("nice_value=-10"), but when entering "startx", just before entering Xfree86, a message is shown: "warning: process set to nice value -11 instead of -10 as requested". If I then change nice value to -11, then the startx script changes nice value to -12. If I change it to -12, it changes it to -13, and so on for any number between [-20, 19]. I've searched for many days (weeks, in fact) on the web and asked at #debian, but no answers. I hope you can lend me a hand. Btw, XFree86 runs perfectly ok, it's just that I hate having error messages. Thanx a lot for your help!

[Thomas] By my knowledge of how X starts up it is not startx which is changing this. Just out of curiosity, run (as root):
dpkg-reconfigure xserver-common
to see if that fixes anything. I have grepped through the startup files that I use (I am running debian unstable) and there is nothing besides the value in /etc/X11/Xwrapper.config which sets or changes the nice value of X.
As a long shot (and possibly a complete aside) you don't have the "and" package installed, do you?


CPU Support on Linux

Sun, 22 Jun 2003 14:10:02 +0100 (BST)
Ashwin N (The LG Answer Gang)
Question by Shishir_Bagchi (Shishir_Bagchi from lko.tcs.co.in)

Upto how many CPUs does Linux Support in --

a) CISC Technology

b) RISC Technology

[ashwin] The numbers for the individual architectures for the current kernels can be got here - http://www.tldp.org/HOWTO/SMP-HOWTO.html
Expect much much better support with the 2.6 series.


mac 5200

Wed, 31 Dec 2003 19:53:51 -0500
Neil Youngman (The LG Answer Gang)
Question by sheldon k feldman (cru1se from comcast.net)

Can I load linux onto this machine? Where do I get the software? thanks

[Neil] MkLinux supports the 5200 see http://linuxtoday.com/developer/2000080401404OSHWKN
Other Mac Linuxes include Yellow dog Linux
http://www.yellowdoglinux.com/support/hardware/breakdown/index.php


Miller-Daemon

Fri, 6 Jun 2003 22:17:25 EDT
Weswwallace (Weswwallace from aol.com)
Answered By Jason Creighton

How do I get an E-maikl to above subject? They interrupt my messages.

[Jason] What you seem to be asking is "why am I getting messages from Mailer-Daemon"? A message from "Mailer-Daemon" is probably the MTA (Mail transport agent: The software that delivers mail.) sending a message that says, in effect, "excuse me, but you seem to be confused. There is no such mailbox here."

claim no such addresses exist, etc.,

[Jason] The automated message you're recieving means just what it says: No such address exists. For example, if you try to send email to bob@example.com, and there is no user bob at example.com, you're probably going to get a bounce message that says no such address exists.

yet they cannot be questioned or challenged ever when they are wrong. Help.

[Jason] The reason "they" cannot be questioned is because you're getting an automated message: It was not send by a human.
Nobody would know better whether an address exists than the system you're sending mail to, so I would say that the system you're sending mail to is right and you are wrong.


what is "no"? (missing prog for binutils compile)

Thu, 29 May 2003 15:53:14 -0700
Faber fedor, Jim Dennis, Alan Mudra (The LG Answer Gang)
Question by Karl-Heinz Herrmann

Now -- it's an unbearable situation that my Linux doesn't know "no".... But I've no idea what it is and you can imagine that a google for "no" even with linux and some other keywords around are not very helpful.

Any ideas?

[Faber] Maybe it's called "nein" on your computer? <grvf>
I can't find a "no" on my Red Hat 8 box either.
[JimD] I think /usr/bin/no was (would be) a counterpart to the old /usr/bin/yes command:
 	#!/bin/sh
	OUTPUT='y'
	[ "$#" -gt 1] && OUTPUT="$*"
	while : ; do
		echo "$OUTPUT"
		done
... so "no" could just be an alias or script that calls /usr/bin/yes with the "no" argument:
		 /usr/bin/yes no
I realize this sounds silly and stupid, and April 1st is long past for this year. But I'm not kidding. That Makefile (or whatever) seems to actually want to pipe an endless stream of "n" or "no" lines into some other process. (/usr/bin/yes was traditionally used in a pipeline with fsck to automate the process of repairing a filesystem that need lots of work -- then they just added the -y option to the GNU/Linux versions of fsck.

I put the question up with bugreports for binutils and got:

[Alan]
> /bin/sh: no: command not found
This is a result of binutils being stuck on using old buggy autoconf. Install a new version of GNU gettext, or configure with --disable-nls.
-- Alan Modra IBM OzLabs - Linux Technology Centre

Got a new gettext which includes some "no"'s

khh > find ./ -name "no*"
./gettext-runtime/po/no.po
./gettext-runtime/po/no.gmo
./gettext-tools/po/no.po
./gettext-tools/po/no.gmo

unfortunately with a new gettext (gettext-0.12.1.tar.gz) and nls enabled I get a linker error for some gettext symbol. The solution without nls works for getting binutils compiled. I try that on the 2.5.70 kernel sometime soon.

Does anybody know of a backport to 2.4.X of the preempt patch and or the I/O scheme patches mentioned on kerneltrap right now? I'm not yet sure what else will break if I switch to 2.5.X. At least NVdriver, lt_serial+lt_modem and vmware kernel modules would be nice to have.


What is Linux Torvalds wife famous for

Thu, 19 Jun 2003 15:00:53 +0100 (BST)
Ashwin N, Thomas Adam (The LG Answer Gang)
Question by Pam Drake (pdrake from northern.wvnet.edu)
[Thomas] Well, she's famous for being married to Linus Torvalds.
[Ashwin] She is also famous for being Finland's champion in kung-fu or some similar martial art ;-)
Linus was interviewed in Issue 67 of Linux Journal, by Marjorie Richardson at the Linux World Expo of that year. Tove was 6 years running, the Finnish champion for karate, specializing in precise Kata (the forms), then moved on to other interests. -- Heather


pppd problem

Thu, 26 Jun 2003 01:30:57 +0530
Joydeep Bakshi (joy12 from vsnl.net)
Answered By Thomas Adam

Hi all,

I use kppp under linux to dial-up my isp. but there is a strange problem happens with me. when I dial-up from windows98 it connects with the isp at the very first attempt. but under Linux (debian woody) kppp takes at-least 3/4 attempts to connect the isp, and during the failure it shows *pppd can't be started * . I have also started pppd from root manually at the time of hooking, but the result is same. could some one please suggest me how to fix the problem ?

thanks in advanced.

[Thomas] You need to ensure that you have a ppp-chat script enabled which is used by pppd to comminicate with the modem to send certain signals, etc.
wvdial will help for this


PPP & ETH won't work concurrently

Fri, 5 Dec 2003 09:48:52 -0800 (PST)
Joe Slobotnik (cj from rt.nl)
Answered By Thomas Adam, Karl-Heinz Herrmann

Hi,

I'm running red hat 9 with an ethernet card to a LAN and a modem for dial up. I'm using kppp for the dial up. When the eth0 int is active, kppp will establish a connection but DNS fails. The DNS listed for both interfaces is the dialup one, but I can't resolve names unless the eth0 is shutdown. How can I , an ordinary joe get these things to work at the same time ?

thanks,
cj

[Thomas] There is either a really easy or hard explanation. I think I am right in saying that you need to have a correct entry in your routing table to use the two concurrently, since the routing will not know.
As far as DNS goes, do you have a valid entry in /etc/resolv.conf and also an entry in /etc/nsswitch.conf:
hosts: files dns
like that? If not, add the "dns" after the word "files".
[K.-H.] Well kppp (or pppd which kppp calls) refuses to setup a default route if one already exists. The magic scrying ball (glass?) would suggest with eth0 up you've a default route set. On dailup via kppp you won't get a default route to your ppp0 interface and therefore DNS lookups to the world outside never reach there.
This is how it should look lie with both eth0 and ppp0 (kppp) up (I cut out three columns which are unimportant):
# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags  Iface
62.104.218.38   0.0.0.0         255.255.255.255 UH     ppp0
192.168.2.3     0.0.0.0         255.255.255.255 UH     dummy0
10.10.10.0      0.0.0.0         255.255.255.0   U      eth0
172.16.57.0     0.0.0.0         255.255.255.0   U      vmnet1
0.0.0.0         62.104.218.38   0.0.0.0         UG     ppp0
I've two private networks, real ethernet as 10.10.10.0 and vmware virtual host as 172.16.57.0. The "0" at the end suggests and a genmask of 255.255.255.0 proves that these are networks, i.e. not a single host but all hosts 10.10.10.x with 1<x<255 (zero is broadcast address to all). If you look at the flags "H" means this is only one single host, "G" means this is a gateway.
The last line is the default routing, i.e. if no other routing rule applies all remaining traffic this way. The way is interface ppp0 and the target the gateway IP 62.104.218.38, our remote host on the other end of the modem line (see first line).
Now I'm pretty sure that in your case there is a line like:
0.0.0.0      [some IP]           0.0.0.0        UG      eth0
If this is the case kppp will not touch it and if you would have looked carefully in your /var/log/messages (or kppp's log) you would have found an error telling you this.
If I guessed right run (as root):
route del default
then start kppp's dialin
After that figure out why RedHat sets a default route (I'm with the lizard and without hats).
If I did not guess right you oviously didn't give enough information....


Red Hat Linux Install issues

Sun, 29 Jun 2003 15:34:59 -0400
Srinivas Velury (s_velury from hotmail.com)
Answered By Mike Martin

Hi Answer Guy,

I am having a peculiar problem with the Red Hat 9 installation. I am trying to install it on a IBM thinkpad 1721 laptop with a formatted hard drive. The laptop has a combo floppy/CD ROM drive. The laptop seems to be seeing the CD ROM dirve but it is not able to read anything off it. The laptop has a PCMCIA network adapter card which I used to connect to my home networking. The other laptop on the home networking has Win 2000 running on it. Here are the following waya I tried to install Linux -

1. I created a Linux boot disk using the pcmcia.img and booted the IBM laptop and selected NFS install. On the other laptop I copied all the Red Hat folders from the 3 installation CDs. I selected "Automatic DNS configuration" to configure the DNS names for the IBM machine. For the NFS server name I typed in the name (which did not have a domain name since I log into a workgroup and not a domain) and typed in the actual directory of the Redhat parent folder (d:\redhat9). But I get an error that the drive could not be mounted.

Then I grabbed a Freeware called NFSClientServer and installed it on the Win 200 laptop. Here I exported the RedHat directory (d:\RedHat9\RedHat). Then I followed the above steps on the IBM machine for NFS installation. Even though my requests form the IBM machine seemed to log in the Server log on the Win 2000 machine, I still could not mount the directory. What am I missing?

2. Failing the above attempt, I attached a SCSI CD ROM to my IBM machine. Then I made a boot disk using the bootdisk.img. After booting the IBM, I selected "local CDROM" for the source, but I kept getting the message "No software found in CDROM". Obvously, Linux was not seeing my SCSI CDROM, but since it detected my internal (failed) CDROM, and could not read off of it, it was giving me the message. How do I make it look at my SCSI CDROM?

I even tried making a boot disk using the "drvblock.img" file, but for some reason when I use the disk, I get the message " No operation system found". When I look at the disk contents the, it seems like the format information on the disk is lost and Windows explorer asks me if I "want to format the disk"? I even tried the same using Linux 7.3 (Red Hat), same error.

Please help.

Regards,
A Wexed Linux Installer.

[Mike] OK - your best bet would be a HD install. If you have copied the folders over you have enough HD space.
In w2k make a directory to hold the iso images In turn copy the cds to this directory (not the contents - I think Nero should be able to do this) You should then have three files in this directory, all ending in .iso
Now boot using your boot.img disk. When you get get prompted for installation type, choose HD then select the right partition where there are held (probably /dev/hda1 on your system if you choose the first partition) then select the directory you have placed the .iso files in.
Then you should be good to go.


pam and ssh

Wed, 25 Jun 2003 10:08:28 -0500
Lou Lohman (LLohman from WeberMarking.com)
Answered By Thomas Adam, Michael Gargiullo

Dear Answer Gang,

Try as I might, I am unable to figure out just what to do to allow a trusted user on a trusted host to use 'scp' to copy files from one system to another WITHOUT a password prompt. We WANT to do this in order to use 'scp' in scripts initiated by CRON. We NEED to do this because 'rcp' gives us 'file too large' responses. I have tried modifying /etc/pam.d/sshd (at least to the limits of my understanding). I can make scp NOT work at all, or require the password, but I cannot make it work without the password. Changing the /etc/ssh/sshd_config file doesn't make any difference, that I can see, even though that file now says 'go ahead and use the rhosts files'.

Can someone help me? Anyone? I would offer a carrot, like "I'll subscribe to LJ", but I already do. I would offer virtual hugs and kisses, but the respondent might be male, in which case an old homophobe like me is going to have a problem. How about "undying gratitude"? Yeah, that's the ticket ... it's easy, it's cheap, I could even teach my kids to sing appropriate praises .. yeah, that's it .... Where's Jon Lovitz when you really need him?

Lou Lohman


Don't just BELIEVE!! Consume information like a starving person, and then sort it out for yourself.
And we know he reads ask-the-gang.html, he gave us explicit permission to publish the whole thread just like we ask for :) Now if we could get people to remember to turn off HTML in their email... -- Heather
[Thomas] You need to run "scp" with the "-B" flag, ie.
scp -B files thomas@thomas
Batchmode doesn't require password authentication. You can also add this in /etc/ssh_config as...
BatchMode=yes
that way, you don't have to pass the -B switch each time.
[Mike] You can also exchange keys from one user/machine to the other. If you want to copy from machine A to machine B. On machine A, as the user that needs to copy, run ssh-keygen -t dsa. This will generate the key pair. Then he'll need to copy the contents of ~/.ssh/id_dsa.pub from machine A into ~/.ssh/authorized_keys on machine B. Then copying from machine A to machine B won't require a password for this user. Do the reverse to copy from B to A.


Pctel modem ................NO CARRIER..........?????

Thu, 26 Jun 2003 08:22:13 +0530
Vivek Ravindranath (vivek_ravindranath from softhome.net)
Answered By Karl-Heinz Herrmann

Hi Answer Gang,

I have some problems trying to connect with my HSP Pctel Micromodem 56.I have RedHat 9 installed and I use kppp for conneting to the net.When I try to connect the handshake goes on well and all of a sudden I get this "No Carrier" message.Can you tell what causes this error and what is the solution.............please sugget an alternative driver if available.At present I am using the Pctel drivers available at www.linmodems.org version 0.9.6 . Can any special AT command string help?????? If yes please suggest a solution.Or using any other dialer will help????

Thanks in advance.

Vivek.

[K.-H.] no carrier means the modem is unable to detect the carrier frequency onto which data would be encoded. As this is a fatal problem it hangs up and tells you "no carrier".
Why is the carrier gone? difficult to tell from here. What do your logfiles say? kppp has a log button, use it!
At exactly what point of connection negotiation does the error occur? Do you get the "connect" in the log window? Then the dialing is finished and control passed to pppd -- which logs it's messages in /var/log/messages or some such place. Go look for it. You can pass additional option to pppd in kppp: add "debug". Make sure to press the add button in the kppp window so the new option is actually used (should show up in the lower larger window).
My guess: serial connection gets established, pppd get's into some trouble negotiating the ppp parameters (user/passwd? pap <-> chap <-> terminal authentification, compression,.... ) and the other side terminates on you. Your modem detects the lost carrier and tells you.


Which VT? Custom prompts.

Mon, 23 Jun 2003 13:59:58 +0100 (BST)
Toby Poynder (toby from whatcouldpossibly.com)

Here's a suggestion for your 2 cent tips section - I hope you like it.

When working with Linux servers I don't run X, preferring to make use of the virtual terminals via Alt-F1, Alt-F2, Alt-F3 etc. The problem is knowing which screen I am looking at, so I have devised a custom prompt by including the following two lines in my .profile:

TTY=`basename \`tty\" export PS1="[$TTY] \w$ "

Cool :)

The first line sets the environment variable TTY to the number of the current virtual terminal, the second sets my prompt to show that number and the current working directory as in this example:

[2] /etc/xinetd.d$

That's it!

Toby Poynder London, UK

I must admit that I often find just typing in "tty" is more efficient than having it set in one's $PS1 prompt. :) -- Thomas


workaround for black lines on Radeon mobility 7500

Wed, 14 Jan 2004 22:06:10 +0000
Adrian Bridgett (adrian from smop.co.uk)

To workaround black lines in XFree86 with Radeon mobility 7500 (thinkpad R40 here), add:

Option ""XaaNoSolidTwoPointLine""

To the drivers section of /etc/X11/XF86Config(-4)

Adrian (aka Wyvern on #hants)


Cool toy of the week: XaoS

Mon, 16 Jun 2003 20:43:40 -0600
Jason Creighton, Faber Fedor, Ben Okopnik (The LG Answer Gang)
[Jason]
http://xaos.sf.net
What is it, you ask?
Real-time fractal zoomer.
If that isn't enough to get you to run out and download it, you're probably not the kind of person who would enjoy it. :-)
[Ben] And don't forget to take a look at the sample fractals that come with the "xaos" package, located (under Debian) in /usr/share/XaoS/examples. (I've got all of these converted to JPGs and use them as randomly-selected desktop backgrounds.) Beautiful.
[Faber] Yes, a very nice toy, but I still miss Fractint for DOS. The X/Windows version had all the features, but just doesn't cut it somehow. A Mandelbrot set not being full screen looses something, not to mention they're slower than the DOS version.
I also had an internal IBM program[1] that would map out different phase space maps stereographically, looking like some of the maps in the Discrete screen saver in xscreensaver.. Man that was a fun toy; I wish I could find the source code (it was floating around here on disk years ago); I'd try to port that to Linux.
[1] For those of you that don't know it, IBM has a a mini Internet in place with their own version of "free software", to wit programs written by Beamers and distributed to other Beamers for fun and collective profit. And since some of those people had multiple PhDs and worked on esoteric stuff, some interesting programs cropped up. It's the only thing I miss from my IBM days.

This page edited and maintained by the Editors of Linux Gazette
HTML script maintained by Heather Stern of Starshine Technical Services, http://www.starshine.org/

Published in Issue 99 of Linux Gazette, February 2004


The Answer Gang

(?) The Answer Gang (!)
By Jim Dennis, Ben Okopnik, Dan Wilder, Breen, Chris, and... (meet the Gang) ... the Editors of Linux Gazette... and You!


We have guidelines for asking and answering questions. Linux questions only, please.

We make no guarantees about answers, but you can be anonymous on request.
See also: The Answer Gang's Knowledge Base and the LG Search Engine


Contents:

¶: Greetings From Heather Stern
(?)Backing up with tar
(?)Hunting for new desktop hardware
(?)Software suspend troubles
(?)Oh-oh. This isn't looking good for the moment...

(¶) Greetings from Heather Stern

Greetings, gentle readers, and welcome once more to the world of the Answer Gang. TAG, we're it :D

I have to confess that I wondered to myself, what should I babble about this time? As I look at the back issues, I notice some interesting statistics... apparently, I shouldn't feel surprised that things here at Linux Gazette are a bit hectic. That's consistent with all our past Februaries. Ain't tradition grand?

February here where I sit, is often considered the month of romance. I have to admit... I (heart) Linux Gazette.

I mean, it's not like I will be buying it chocolates or wondering what its favorite cologne is. But I put a bunch of work in every month - I really like knowing you people out there are reading (hint, hint; tell us what you like in this stuff)! I love seeing what sort of curious troubles are out there to solve - the most curious, the kinds of things whose answers change over time. And it's always worth seeing what cool answers are out there.

You don't have to actually be a member of the Answer Gang to send us juicy answers, either. An honorary Answerbubble to folks who send us their good stuff - or cc us on the the juicy tip they are sending to someone anyway. When we got the Gang together, the heading mentioned a few of the active posters... and you. That hasn't changed. It's teamwork that makes this all happen, and I'm proud to be here. Thomas and I have teamed up to bring you the juiciest threads this month, and I hope they make your time with Linux this month - "just a little more fun!"


Copyright © 2004
HTML script maintained by Heather Stern of Starshine Technical Services, http://www.starshine.org/

Published in Issue 99 of Linux Gazette, February 2004


Lightweight, (Almost) Crypto-Free Remote System Operation

By Ray Ingles

"There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies, and the other is to make it so complicated that there are no obvious deficiencies." - C.A.R. Hoare

"Sure I'm paranoid, but am I paranoid ENOUGH?" - Unknown

Introduction

System administrators frequently want to be able to work on the machines they run even when they are far away from them. There are secure tools that allow full remote shell access, like ssh and lsh, but due to their complexity they have suffered critical exploits from time to time. In addition, their overhead can be excessive for some purposes. Fortunately, other options are available that can be used alone or can be combined with remote shells to create a more secure overall system.

Overview

Maybe the pager has just gone off when you're home in bed, and the boss wants you to fix the broken database now. Or perhaps you're out for lunch and someone calls to tell you the mailserver has been cracked and is currently spamming the world, and you need to bring it down fast. Possibly you've checked and your Web server has wedged itself and needs to be restarted. Or suppose you're just on vacation and find you want to update your home Web site with some new photos. In all these cases, you'd like to do something to the machine over the Internet without having to actually sit in front of it - things you don't want just anybody to be able to do.

The Problem

Tools like ssh and lsh are great for allowing secure remote access to your system. They offer essentially full, flexible remote control of a machine, in an encrypted and authenticated manner. But they are complex pieces of software; there's no way to do what they do without being complex. And with complexity comes bugs. SSH and lsh, and related tools like Webmin, have all had serious flaws that would allow an attacker to get full control over your system. Leaving them available all the time is a risk - sometimes it's necessary, but it's still a risk. And in some cases, you'd like to be able to tell the machine to do something, but it's not even attached to the network on a regular basis.

Some Solutions

It would be nice to enable remote shell access only when necessary. And perhaps (for something like shutting down a mail server) you don't even need a full shell, just a way to fire off a script remotely. Of course, the problem then becomes, how do you know that the alternative software is any more secure than ssh itself? Various people have worked on this problem in the past, and several potential solutions are available, ranging from the simple and venerable to the new and exotic.

Xringd uses a modem to control a machine remotely. Mail filters can be used to trigger actions based on special messages. Some solutions (like 'port knocking' and 'Net::Pcap') use the network, but without requiring even a single open port. Lando runs commands over a network, using username and password. Most recently, a program specifically for secure remote execution called Ostiary has been developed.

The Options

Old School Remote Operation - Xringd

The eXtended Ring Daemon, or "Xringd", uses a modem to monitor rings on a phone line. It counts the number of rings, and the time between them. If a 'sequence' matches one of the ones that it has been set up to detect, Xringd will run an associated command.

This is very nice from a security perspective. Since it uses no network connection at all, it's entirely immune to network attacks like buffer overflows. It can be used even when a network connection is unavailable (it's often used to cause a computer to initiate a dialup connection). The only 'client' you need is a phone. If you use it to start up ssh on demand, then the attacker needs to know the right phone number and the right ring pattern - it's quite hard to sniff that kind of thing remotely. It's also highly resistant to a man in the middle attack. (If you have to worry about someone rerouting your phone calls, you're in more trouble than Xringd can save you from.)

There are some practical issues that may make this unattractive in some circumstances. You need a modem and a telephone line to the server. (Fortunately, you don't need a fast modem at all; even a 1200 baud one will do nicely, but some servers are not placed close to a telephone jack.) Also, things like answering machines or voicemail (or even other people answering the telephone) can interfere with Xringd. If you give the server a dedicated line, you can avoid these problems, but that can be costly.

Finally, note that the rings you hear when making a call are not necessarily synchronized with the ring signals actually sent to the telephone. In most circumstances, they are close enough, but reliability can be an issue at times.

Procmail And Other Mail Filters

Most of the mail filtering programs have a way to invoke scripts when mail matching a pattern is received (in the simplest case, mail to a particular address). Assuming the server is running an SMTP daemon, this can be a nice way of triggering actions remotely. Technically, one could even send a shell script to be run, and have it e-mail the results back to you, giving you the equivalent of a very slow remote shell. The only client needed is an e-mail program, or even a webmail account.

The first problem is that if the box you want to talk to doesn't accept e-mail, this obviously won't work. (Adding an entire mail server, with the attendant risks of bugs, spam load, etc., just for remote execution doesn't make a lot of sense.) Some machines only periodically collect e-mail from a primary server, so there can be a substantial delay between when a command is sent and when it is acted upon.

Furthermore, if you don't encrypt the traffic in some way (or at least sign it with PGP), then anyone sniffing traffic between you and your server may be able to take advantage of the same channel to do mischief, or perform a man-in-the-middle-attack. (E-mail traffic is notoriously easy to falsify; hence the avalanche of spam these days.)

CVTSA, or "ClairVoyanT SysAdmin", is a system designed specifically for running commands through e-mail. It has some support for using passwords, but does not (currently) encrypt them in transit, so a sniffer could capture them and use them again.

Of course, if the only things you want to do with this type of system are emergency shutdowns and other such (hopefully rare) crisis management, then even an unencrypted channel might work. However, you'll need to change the 'magic trigger pattern' each time after you use it, or you take the risk that an attacker might capture it and 'replay' it at an inconvenient time.

Port Knocking

With port knocking, a daemon monitors firewall logs, looking for particular sequences of connection attempts to particular (closed) ports. When it sees a sequence it recognizes, it runs the associated command. This isn't terribly bandwidth efficient, but it has some nice properties. First, it's hard to tell if a server is listening for port knocks. Second (and most important), it's awfully hard to crack a closed port. (Linksys routers have had a simple version of this for a while, BTW, that they call port triggering.)

However, a clever attacker with a sniffer could notice this traffic, and duplicate it for their own use. More complicated encodings could express something like a PGP signature (indeed, in theory one could create an entire network protocol based on port knocks), but things rapidly become difficult to work with. As with 'mail filtering' solutions, one can either use it sparingly in emergencies, or move to real cryptography.

It's also important to realize that this system is critically dependent on the probe packets actually being delivered, and delivered in the order that they were sent. This is not guaranteed on the Internet. What's more, depending on where you're at (e.g., an Internet cafe or behind a business firewall), you might not be allowed to connect out to arbitrary ports. The more complex you make the 'knocks', the less reliable the system will be.

Also, notice that at least one entire IP packet (28 bytes or so minimum) is used to transmit roughly one bit of information. In terms of network efficiency, it's almost hideous. For a simple 'open up ssh' message, it's not a consideration, but actually adding cryptographic security to this system could use up a decent chunk of the available bandwidth.

Finally, this increases the CPU load for each entry in the firewall log. Depending on how detailed the logs are, and how fast and busy the network is, this can be a significant drain on resources.

'Sniffer' Triggers - libpcap and friends

Another interesting approach is to use Net::Pcap or other network capturing software to look for specific packets on the network (e.g., DNS requests) and examine them for particular data (e.g., a particular address). If found, it can enable ssh temporarily, or perform other actions.

One potential benefit of this approach is that a computer doesn't have to have an address on a network in order to monitor traffic on that network. You can set the card to 'promiscuous mode' and examine all the traffic on the wire. (It's very hard to hack a machine you don't even know is there.) Once the 'trigger' is spotted, the sniffer can use other means (a separate network, a serial link, even Xringd) to open up SSH on a target machine. Of course, you can also simply run the sniffer directly on the target.

Again, a clever attacker with their own sniffer may be able to detect the unusual activity and correlate it. To make this system truly secure, you would need more complex encoding/encryption of the 'trigger' traffic.

Additionally, the CPU load for this solution can be even worse than for 'port knocking' systems. A 'port knocking' daemon monitors firewall logs, which can have variable levels of detail. By necessity, a 'sniffer' solution must examine every packet on the network segment, which can be a substantial task for a busy gigabit line.

Lando - Simple & Flexible

Lando allows a user to run a preconfigured set of commands remotely, using passwords, and even allowing the user to supply arguments to them. While it currently has only a Windows client, and passwords are sent in the clear (making it suitable only for use on a trusted local network, or perhaps on a VPN), it can be very useful for, e.g. operating a local firewall box without going to the trouble of logging in.

Okay, A Little Crypto - Ostiary

All of the above solutions have their advantages, but each has some practical issues that can make them unsuitable for particular applications. Ostiary was designed to be a secure alternative that uses minimal resources. It tackles this problem with what might be termed "aggressive simplicity". It does require an active connection to the network (unlike Xringd and sniffing), but allows for much better default security with very low CPU, RAM, disk, and network bandwidth requirements.

An Ostiary server has one open port that it listens on. When someone connects, the server sends a random fixed length 'salt' message 16 bytes in size - the size of an MD5 hash. It then waits (with a timeout) for a reply from the client. It reads (at most) 16 bytes of reply, and closes the connection.

Ostiary has a list of commands to run, with associated passwords. It runs through the list, and hashes these passwords with the 'salt' it sent to the client. If one of these hashes matches the reply from the client, the associated command is run. (One final touch is that a record is kept of connections, and clients with too many failed attempts are 'locked out', and all subsequent communication from them is ignored.)

A detailed security analysis is available, but a few things about this system should be clear. With a protocol this simple, the chances for dangerous bugs are drastically reduced. Using fixed-length messages essentially eliminates the chances of a buffer overflow or other memory error. (Indeed, Ostiary does no dynamic memory allocation of any kind - everything is stored in static, fixed-size data structures.) Replay and man-in-the-middle attacks are also effectively useless. Ostiary limits how fast it accepts connections, enforcing low CPU and network usage. (The first production Ostiary server was a 16MHz 68030 machine.) Client requirements are even lower: Clients are available for Palm Pilots and even Windows.

Unlike a procmail-based solution, where you can put arbitrary commands (with arguments) in the message, Ostiary can only run the fixed set of commands you have preconfigured. The only argument it supplies to the commands is the IP address of the client that initiated the command. It requires an active network connection (unlike Xringd) and an open port (unlike port knocking or sniffing), which may entail configuring a firewall to open a new port. (Although one could run Ostiary on, say, port 22, and upon receipt of the correct command, it could terminate itself and spawn sshd...)

Since Ostiary uses TCP, it is as reliable as the network it uses to communicate. Problems with miscounted phone rings (a la Xringd) or randomly dropped packets (a la port knocking) are not a concern.

Summary

The following table summarizes the pros and cons of the various systems outlined above. "Replay" and "Man-in-the-middle" indicate if the default system is vulnerable to the corresponding attacks. "Command arguments" indicates if the system can run arbitrary commands with arguments. "CPU load" indicates that CPU time can be a significant consideration. "Special client" indicates that a specific client program is needed to work with that system.

SystemXringdMail filterPort knockingSniffersLandoOstiary
Network Required? YesYesYesYesYes
Port Required? Yes YesYes
Modem Required?Yes
Replay? YesYesYesYes
Man-in-the-middle? YesYesYesYes
Command arguments? Yes Yes
CPU load? SometimesYesYes
Special client? SometimesSometimesYesYes

None of these approaches is right for everyone. But all of them can be used to make attacks at least more inconvenient, and in many cases far more difficult. Remember, though, to analyze their pros and cons relative to your specific situation. Also remember that true security is a process, not a goal - you can never just install some software and be done thinking about it.

[BIO] Ray Ingles has been involved with Linux since 1995. In addition to being an active member of the Metro Detroit Linux User's Group, he has made minor contributions to the UPS HOWTO and the Linux Joystick Driver.

Copyright © 2004, Ray Ingles. Copying license http://linuxgazette.net/copying.html

Published in Issue 99 of Linux Gazette, February 2004


XMLTV

By Bill Lovett

Where do you go to find out what's on TV? The usual suspects might include a newspaper, a recent issue of TV Guide magazine, a favorite Web site, or your nearest TiVo, ReplayTV, or other PVR. But don't forget to add Linux to the top of that list. You can let the machine do the dirty work and bring the listings to you. XMLTV, a short bash script, and a cron job are all you need to get started.

Installation

First things first: getting the program installed. XMLTV is a suite of Perl scripts and can be downloaded from membled.com/work/apps/xmltv. There are releases for Unix-like and Windows environments, but, for obvious reasons, we'll focus on the former. If you're installing from source, it's the usual routine:

% perl Makefile.PL
% make
% make test
% make install 

If you're on Debian, it's all just an apt-get away (apt-cache search xmltv). Links to packages for OS X, Red Hat 8, and Red Hat 9 are available from the project's homepage.

Configuration

Before XMLTV can be useful, it needs to know where in the world you are. XMLTV is international— it can fetch TV listings for Canada and the United States, the United Kingdom, Austria and Germany, New Zealand, Finland, Italy, Spain, the Netherlands, Denmark, and Hungary. (Belgium and France are in the works.) The scripts that collect listings for a particular country are referred to as grabbers, and you'll find them on the command line under tv_grab_*. We'll use the U.S. grabber, tv_grab_na.

When you first run the grabber, do so with the --configure option. This starts a question-and-answer session in which you and the grabber get a little bit better acquainted, as far as your Zip code, TV service provider, and channels you want to ignore are concerned. The results of the script are written to ~/.xmltv/tv_grab_na.conf, and can be easily edited by hand.

At this point, XMLTV is ready to do your bidding. Do a man tv_grab_na to learn about all the available options. For now, just two will suffice:

% tv_grab_na --days 1 --output /tmp/tv.xml
This tells the grabber to get one day's worth of listings, and save them out to /tmp/tv.xml.

XMLTV's file format doesn't quite make for friendly reading, unless you enjoy reading raw markup. A few more scripts from the suite can fix that. tv_sort sorts the contents of an xmltv file date. tv_grep lets you weed out some of the obvious cruft in the listings. Here's how I run it:

% tv_sort --output /tmp/tv_sorted.xml /tmp/tv.xml
% tv_grep --output /tmp/tv_grepped.xml --ignore-case --not --category Children \
          --not --category Sports --not --title "Paid Programming" \
          --not --title "Local Origination" \
          --on-after now /tmp/tv_sorted.xml
The commands above sort the original file and then discard anything categorized as "Children" or "Sports", and anything with "Paid Programming" (infomercials) or "Local Origination" (public access) in the title. Also, we're discarding everything that aired before the script ran.

At this point, we've still got an XML file. Converters to the rescue! tv_to_text is one of the tools that can help us go from XML to something else. (Other possibilities include LaTeX, HTML and PDF. Check the readme to see what's currently available.) After running something like this:

% tv_to_text --output /tmp/tv.txt /tmp/tv_grepped.xml
We get output like this:
21:00--21:30    Spy School      38
21:00--21:30    Designing for the Sexes // European Kitchen     64
21:00--21:30    Chappelle's Show        67
21:00--21:30    The Real World // Las Vegas     71
21:00--22:00    Law & Order: Special Victims Unit // Guilt      44
21:00--22:00    Wild Card // Auntie Venom       45
21:00--22:00    Cold Case Files // The Accidental Killer; Little Sister Lost    57
21:00--22:00    America's Most Wanted: America Fights Back // Top Ten Most Wanted Fugitives     5
21:00--22:00    The FBI Files // The Price of Greed     60
21:00--22:00    Trading Spaces // Nashville: Murphywood Crossing        61
21:00--22:00    Great Chowder Cook-Off  63
21:00--22:00    Ends of the Earth // Secrets of the Holy Land   65
21:00--22:00    The E! True Hollywood Story // The Hilton Sisters       68
...
Simple and no frills. Just what we need for the final step: e-mail delivery.

Delivery

If we stopped at this point we'd have used several of XMLTV's abilities but hardly anything else. We'd also be running low on convenience and automation. Fortunately, we can wrap all the commands we've seen so far into a shell script, and have it e-mail us the final results. mail can take care of, well, the mailing:

% mail -s "Today's TV listings from XMLTV" user@localhost < /tmp/tv.txt
Here's what the full script looks like (text version of this listing):
#!/bin/sh

# Grab today's listings:
tv_grab_na --days 1 --output /tmp/tv.xml

# Sort
tv_sort --output /tmp/tv_sorted.xml /tmp/tv.xml

# Grep
tv_grep --output /tmp/tv_grepped.xml --ignore-case --not --category Children \
--not --category Sports --not --title "Paid Programming" \
--not --title "Local Origination" \
--on-after now /tmp/tv_sorted.xml

# Convert To Text
tv_to_text --output /tmp/tv.txt /tmp/tv_grepped.xml

# Email
mail -s "Today's TV listings from XMLTV" user@localhost < /tmp/tv.txt

Put that in a cron job that runs once per day, and you've got TV listings with no outside advertising, and no channels or shows you know you aren't interested in.

More importantly, you've got a foundation to build on. What we've covered is just the beginning— beyond the command-line scripts, a GUI client is also available. Of course, there are plenty more things you could do from the command line, such as:

  1. Pull in data from imdb.com via tv_imdb
  2. Split the listings into separate files for each day and channel via tv_split
  3. Transform the XML with your own XSLT stylesheet.
  4. Only send e-mail if certain keywords are found
It all depends on how you want to consume the information, and how cleverly you can chain all the scripts together.

[BIO] Bill Lovett is a Web developer in New York City. He's one of those PHP/MySQL types. And he has this weird thing about running Linux on old machines that by all rights should have been trashed years ago. Read more about Bill and his Open Source projects at www.ilovett.com


Copyright © 2004, Bill Lovett. Copying license http://linuxgazette.net/copying.html

Published in Issue 99 of Linux Gazette, February 2004


Let's Build a Cool Linux Toy

By Pramode C.E

Many of us make a living out of Linux - but, if somebody asks us why we are so crazy about it, one common answer would be `fun'. Playing with Linux is lots of fun - with the added benefit that, most of the time, you end up learning a lot. Recently, I happened to come across a nice book which tries to emphasize the `fun' aspect of Linux - it describes several small `projects' (a jukebox, a picture frame, etc.) that a moderately experienced Linux user may be able to implement on her own. One of the projects involved interfacing with a temperature-sensing element and putting up the temperature value on a Web page (or including it in your email signature - and any other crazy stuff which you can imagine!). The only trouble was that, in the part of the world where I live, walking up to an electronics store and asking for an integrated, 1-wire temperature-sensing element is more likely to yield a hard stare than anything else. Smart sensors that can be directly interfaced to the PC with the minimum of fuss are seldom available off-the-shelf - you will mostly have to `roll your own' - which adds to the fun and excitement. With a low-cost general purpose microcontroller like the PIC16F628, bits and pieces of cheap, commonly available electronics components, and LOTS of code, you can build many interesting `toys' and hook them up to your Linux machine - a really great learning experience for the hardware hacker who wants to learn Linux, or the Linux hacker who wants to learn a bit of hardware. This article describes how I went about building my temperature-sensing project - amateur Linux/hardware hackers might find some of the ideas useful when they start building things on their own.

Get a PIC micro, and set it up to work with Linux

This is the first step. Microchip PIC controllers are commonly available. If you are like me, working with a soldering iron for more than 10 minutes would drive you crazy - so you have to choose the right kind of PIC - the one that can be programmed with the simplest possible circuit (connected to the PC parallel port), preferably with a 5V supply. Look no further than the PIC16F628. This is a cool device that has lots of peripherals (except the ADC - but then, we can roll our own crude analog-to-digital converter with the comparator and pulse width modulation facilities offered by the PIC) and supports a `Low-Voltage Programming Mode'. I found a nice little circuit (the simplest circuit, and one that works perfectly, out of the dozens I have seen on the Net) designed by Jim Paris for a microcontroller programming laboratory at MIT. Here is the circuit:

[diagram]

I assembled the circuit on a breadboard for testing in a few minutes' time.

Jim Paris has designed a program (called `jimpic') for burning machine code onto the flash memory of the microcontroller. It is available for download from here. I wrote a simple assembly language program, converted it into machine code with the help of the `gpasm' assembler available as part of the GNU PIC Utilities Project and burned it onto the micro by running `jimpic' with the `-b' option.

A Quick Introduction to PIC programming

A nice thing about the PIC is that, if you have some background in general microprocessor architecture and assembly language programming, you can become productive with it in just about one or two hours' time. The instruction set is very compact (35 instructions) and sufficient for most simple bit-twiddling tasks. The PIC16F628 packs a decent 224 bytes of data memory with 2K of program (code) memory. The peripherals include general-purpose digital I/O ports, three timers, two analog comparators, on-chip voltage reference module, Universal Synchronous-Asynchronous Receiver Transmitter (for serial communication), and Capture-Compare-PWM module. Special CPU features include a watchdog timer, brown-out detect circuitry, and an internal RC oscillator (so that you won't be needing an external crystal if you aren't too concerned about precise timing).

The general purpose data RAM begins at address 0x20 (the locations below this are Special Function Registers - basically memory mapped I/O ports, control registers etc.). Here is an elementary assembly language program, which simply stores the value 0 into the accumulator (the `W' register, in PIC terminology).

[Listing 1]

(Remove the .txt extension if you download the listing. It's there only to ensure browsers display it properly.)

We will now assemble the file:

gpasm -a inhx8m a.s

The result is an Intel hex format file, which can be given to `jimpic' for burning. Each line of the hex file contains a few bytes of machine code, the address at which the machine code is to be stored (in the flash memory of the microcontroller), some kind of checksum, and some other information. Here is the hex file generated by running `gpasm' over our assembly language program:

:020000000030CE
:02400E00983FD9
:00000001FF

The first line of our program tells the assembler that machine code is to be generated for the PIC16F628. The second line includes a file (available with the `gputils' distribution) that contains lots of symbol definitions. The third line, a __CONFIG directive, tells the assembler what special features of the microcontroller (say, the Watchdog timer) should be enabled/disabled by writing bit patterns to a `magic' `configuration word' within the PIC; _WDT_OFF means we don't want the watchdog to be enabled, _INTRC_OSC_NOCLKOUT means we are going to use the internal oscillator to provide the timing signals necessary for program execution. You will have to refer to the 16F628 datasheet to know more about these configuration bits. The fourth line is the only proper assembly language instruction in the program - it moves the `literal' (constant) value 0 to the `W' register. Note that each line begins with a tab.

Lighting up an LED

Here is a program that lights up an LED connected to the RB0 pin of the microcontroller:

[Listing 2]

PORTB is an eight-bit port - the direction of each pin (i.e., whether the pin is to act as input or output) is controlled by individual bits of the TRISB register - if a TRISB bit is set, the corresponding PORTB pin is input - otherwise it is output. The PIC has the concept of `banked' addresses, which is rather confusing to the beginner. (It's a headache even if you are an `experienced' developer.) You visualize `banks' of special function registers - the STATUS register is the same across all the banks while the TRISB register is available only in bank 1. You are by default in bank 0. To access TRISB, you have to `switch over' to bank 1. This is by setting the RP0 bit of the status register. (When you read microcontroller manuals, you will see that not only are the control registers given special names, even the individual bits are named. Header files available with the development kit for the microcontroller map these symbolic names to the numbers given in the manual, making the life of the assembly programmer a bit easier.) The `bsf' instruction (bit set f - `f' represents the fact that the number that comes as the operand for the instruction represents a memory address or a special function register and not a `literal') takes two operands - the first one being the address of a RAM location or a special function register, and the second, a bit number. The `movwf' instruction copies the contents of the `W' register to the memory location whose address is the operand of the instruction.

Building a `running' circuit

After assembling and burning the above program, we are ready to see it in action. The running circuit can be built in a jiffy - place +5V on the VDD pin of the PIC (pin 14), connect Vss (pin 5) to circuit ground, connect MCLR (pin 4) to +5V through a 2K resistor, connect the LED between RB0 and Gnd with a current limiting resistor of say 1K in series - and that's all. You should see the LED lighting up as soon as you apply power. Your next attempt will be to make the LED blink - for that you will have to read a little bit more about the PIC instruction set - the manual will come in handy at this juncture.

Debugging tips

Here are some things that I have found handy while debugging:

The temperature sensor interfacing project

The LM35 is a commonly available calibrated temperature sensor that converts temperature (in degrees Celsius) to voltage - each degree rise in temperature results in 10mV rise in output voltage. It's a three-pin device - Vcc, Gnd, and voltage output. You can get the datasheet from here. Say the current temperature is 23 degrees Celsius; the voltage output would be 230 millivolts.

The question is, how do you convert this voltage to a digital value? The easiest way would be to use a commodity analog-to-digital converter, and interface it to the printer port. Another solution would be to use a PIC with a built in ADC (say the 12F675). The third would be to use some of the peripherals available in the 16F628, write some code, and build a crude ADC of your own. As I had explored the first two options a lot in the past, I thought of trying out the third one.

Two peripheral features of the PIC are of interest to us here - one is the builtin PWM module (Pulse Width Modulation), which is capable of generating, in, hardware, a continuous stream of digital on-off pulses whose duty cycle can be varied simply by storing certain numbers in specific special function registers. Once the PWM module is initialized to generate a pulse train of a specific duty cycle, it will keep on doing so without any software intervention - our program can do something else.

The PIC is also equipped with two analog comparators, which can be configured in a variety of ways. Let's say we are using just one of the comparators. Two PORTA pins can be programmed to accept voltage levels and transmit them to the Vin+ and Vin- pins of the comparator. The comparator output is high if the Vin+ voltage is greater than the Vin- voltage, and low otherwise. The output can be made available on another PORTA pin, or it can be simply read from a particular bit of the Comparator Control Register, CMCON.

Filtering PWM pulses

The figure shows a PWM pulse (off OV, on +5V) of period T being fed to an RC circuit (R*C >> T). If the on-off periods are equal, the output seen across the capacitor will be a constant DC level of magnitude 2.5V. Electrical engineering text books should give you the reason why it is so - or, if you are not very sure of the math involved (as I am), play with some R and C values until you get the desired effect. Now what if you feed a PWM pulse whose on-time is less than T/2? You will see that the output is again a DC level, but the magnitude has come down proportionately. What if you increase the on-time? Again, the output is a DC level, only thing is the magnitude has increased proportionately. Now you have a cool way to implement a DAC, a digital-to-analog converter. Say you want to generate a voltage of 0.449V. What if you program the PIC so as to generate a PWM pulse train of period 256 microseconds and on-time 128micro. The output voltage would be 5V*(128/256.0) = 2.5V. Now, what if the on-time is 23 micro seconds? The output is 5V*(23.0/256) = 0.449V. (I use Python to do these quick-and-dirty calculations. It's one of my favourite uses of this great language.) The on-time can be altered easily by writing some numbers to two registers, CCPR1L and CCP1CON. A pure digital way to generate analog voltage!

From DAC to ADC

What has all this got to do with converting the LM35 sensor's analog voltage output to a numerical value? Well, a DAC, together with a comparator, builds up an ADC. How? Say the maximum and minimum temperatures at your place of residence can never go above/below 45 degree Celsius and 20 degree Celsius. So the sensor's output will always be between .45V and .2V (remember, 10mV per degree change in temperature). We start generating a PWM signal of period 256 microseconds. The RC-filtered output is fed to Vin+ of the comparator, and the sensor's output is fed to Vin-. Let's say the sensor output is .3V. If the PWM on-period is 23 microseconds, the filtered DC level would be 5*(23.0/256) = 0.449V. So, Vin+ is greater than Vin-, and the comparator output (as obtained from a bit of the CMCON register) is high. Now, we start bringing down the on-time. At a certain point, Vin- will go above Vin+, and the comparator output drops to zero. The magnitude of the on-time at this point is a true representation of the analog value of the sensor output. We communicate this number to a program running on the Linux box through a serial link. You can download the PIC assembly language program that does all these tricks:

[Listing 3]

Instead of performing a `linear' search from the high boundary down to the lower one, we can think of generating a voltage that lies in the middle of this range and comparing it with the sensor output. If the comparator says that the sensor output is higher, we can repeat the same procedure on the upper half. This is the classical `binary search' applied to solve a hardware problem! Horowitz and Hill, in their book The Art of Electronics, have oscilloscope traces of this binary search in action! Computer programmers should always show a good amount of skepticism towards code that claims to do binary search - the algorithm looks deceptively simple - but is in fact not very easy to implement correctly.

Back to Linux

The PIC micro sends the temperature data it has gathered out through a port pin (RB2) in a serial manner - this port pin is directly connected to the receive pin of the PC serial port. What remains is to write a program that will read this data and process it in some manner. Even though the RS-232C serial communication standard defines an `on' voltage to be between -3 and -12V and an `off' to be between +3 and +12, I have been able to get satisfactory results using the 0 and 5V logic outputs from the PIC port pin - if it doesn't work out for you, you will have to place a device like the MAX232 between the PIC port pin and the PC serial port receive pin.

Interfacing with the serial port

Let's look at the simplest way to interface an external circuit to the serial port. (We won't be sending any data out through the PC serial port - that would make the circuit a wee bit more complex.) Pin number 2 of the 9-pin PC serial port connector is the receive pin, 3 the transmit pin, and 5, Ground. Let's say the PIC is sending data out through its RB2 pin at 9600 bits per second, 8N1 (8 data bits, no parity, 1 stop bit) format. The UART that controls the PC serial port should be programmed for this particular baud rate and data format. This can be done by writing magic bit patterns to certain control registers. Once that is done, our program can keep on polling a bit of the UART status register to know whether a new data byte has arrived. Here is the code listing:

[Listing 4]

The program has two disadvantages. One, it is using low-level I/O calls, which, if they are to work properly, should be preceded by an iopl() call. Only the superuser can call iopl() successfully - so the program should run under root privilege. We are wasting CPU time when we keep polling for data in a tight loop; that's another big problem. Both are solved by not directly interacting with the hardware - we can make use of system calls to talk to the serial driver within the Linux kernel - which will do all the low level stuff needed to manage blocking, interrupt driven I/O.

Serial I/O in Python

The Python `termios' module can be used for doing serial comm at a higher level. Isaac Barona Martinez has written a simple wrapper over `termios'. It is called uspp and is available for download from here. Using this module, reading from the serial port is a breeze:

[Listing 5]


from uspp import *

# COM1 is initialized at 9600 baud. The
# default data format is 8N1

s = SerialPort("/dev/ttyS0", None, 9600)
s.flush() # discard unread bytes

print ord(s.read()) # s.read() returns a one-character
                    # string. We convert it into its ascii
                    # value

Once you get this far, let your imagination run riot!

Acknowledgements

Thanks to Christopher Negus and Chuck Wolber for a really cool book!

Thanks to Jim Paris, Ariel Rodriguez and Sheldon Chan for the excellent `jimpic' hardware and software. As I had mentioned earlier in this article, I find it to be the easiest way to get started with PIC programming under Linux. Thanks to Isaac Barona Martinez for uspp, which simplifies serial programming a lot.

Conclusion

There are two excellent documents that describe serial programming under Linux. One is the Serial Programming HOWTO. The other is Serial Programming guide for POSIX operating Systems. The Microchip home page contains lots of application notes, reading which might give you ideas for your next Linux hardware hack - just don't forget to share the fun with LG readers! I can be contacted via my home page at pramode.net.

[BIO] I am an instructor working for IC Software in Kerala, India. I would have loved becoming an organic chemist, but I do the second best thing possible, which is play with Linux and teach programming!

Copyright © 2004, Pramode C.E. Copying license http://linuxgazette.net/copying.html

Published in Issue 99 of Linux Gazette, February 2004


OCaml, an Introduction

By Jurjen Stellingwerff

Object Caml is an ML type of language. For the non-gurus: it's a functional language that can also be programmed in a non-functional and object-oriented way.

This language is really easy to learn. It's powerful and keeps impressing me with its speed. Programs written in this language are almost always stable by default. No segmentation faults, only occasional unending loops for the programmers that still hang on to program their own loops. It is really not needed to write most loops, since the libraries contain standard functions that are good enough in 99% of the cases. So try to use those functions: It really pays off in terms of stability of your programs, and, unless you have intimate knowledge of the inner works of this language, they tend to be better optimised.

The language can be obtained from the website caml.inria.fr. Here, they provide RPMs for the RedHat 7.2/8.0/9 and Mandrake 8.0 distributions. Also MS Windows binaries are available, but not all Unix library functions will work there, for some mysterious reason. The source tarball does compile flawlessly for me. It just has a somewhat unusual makefile layout:

# ./configure
# make world; make opt; make install

The normal libraries include many usable data-structures like balanced trees, hash tables, and streams. Their version of header files (.mli files) contain all the basic documentation you need, and those are directly converted into HTML and published on the Web in their OCaml manual. This manual is not very usable to study this language, so I'll try to explain here some of the basic language constructions. This is just to give you an impression of the power of this language.

Modules & Functions

Now some real life examples. I wrote a program to help administrating a computer. It is a subset of a normal file finder, but is a command line tool and very fast. It helps locating large, not-recently-used files to be deleted from the system. It crawls through the directory tree and show the contents in different layouts.

Every module in OCaml has its own namespace. Specific definitions can be found by adding the module name, with the first character an upper-case character. You can also change the namespace of the current program to include a total module. Normally, only the standard module 'pervasives.mli' is included in the default namespace. The example program 'show.ml' starts with:

open Basics
open Unix
open Unix.LargeFile

This includes my own set of 'basics' functions and 2 standard libraries: 'Unix' and 'Unix.LargeFile'. A module normally consists of 2 files. The first file for exporting definitions 'module.mli' (like the C .h file), and the second one for actual code (the 'module.ml' file). The program uses the function 'string_sub' that provides a foolproof version of the 'String.sub' standard function (from the string.mli module). The basics.mli file contains the lines:

val string_sub: string -> int -> int -> string
(** Get the sub string from a [string] from position [from] with [length].
This is the same function as String.sub, but it will never raise an exception.
And a negative [from] value is counted from the right side of the string. *)

This gives the definition of this function and the description. There is an automatic documentation generator (ocamldoc) that reads .mli files and writes .html files as basic interface documentation. Normal comments start with (* but the documentation generator only writes comments that start with (** to the .html files. This document contains links to the documentation of the used modules. This documentation is really helpful to start programming ocaml. The .mli files are all included in the distribution, but the complete manual and a book can be downloaded from the Web site caml.inria.fr

The function is followed by its type. It wants 3 parameters and provides a string. Normally we need to write 'Basics.string_sub' to use this function. But after the 'open Basics' instruction just 'string_sub' is enough.

Basic operations and function calls

Now, back to the main program again. The first function is 'gettype'. It will try to return the type of a file. The file type is defined as the part of the filename following the last '.'. When there is no dot, the type is unknown and returned empty.

let gettype file =
try
let pos = String.rindex file '.' in
String.sub file (pos+1) (String.length file-pos-1)
with Not_found -> ""
;;

This function only uses standard functions. First, it catches the Not_found exception in the 'try' 'with Not_found -> ""' code. All other exceptions will be passed to the caller to be handled, and can possibly stop the main program. The local variable pos get is filled with the result of the function rindex. This function is also the reason to catch the exception; otherwise, the main program might stop on the first found file with no '.' in it. Local variables can be declared everywhere inside ocaml with 'let <variable> = <value> in <code>'. After the completion of the given code, the variable is out of scope and will be forgotten. The data will be passed to the garbage collector to be removed from memory. Function calls do normally use brackets. The function call to 'String.sub' gets 3 parameters the string 'file' the integer '(pos+1)' and the integer '(String.length file-pos-1)'. The last parameter calls the function 'String.length' with a single parameter 'file'. So, the functions are eager for their parameters; brackets are needed only when the parameters are filled with calculations.

Also '(+)' and '(-)' are functions of the pervasives module. It is very easy to define your own operators; just add brackets around their definition, and they are ready.

If then else

The next routine 'filesize' in the example code is far longer, but largely introduces sub-functions and 'if <bool-expr> then <expr> else <expr>' statements. This function creates a string from an int64 number for human readable file and directory sizes. The types of parameters are normally not given; they are determined by ocaml through their usage. When something is not clear, the compiler or interpreter will complain about it before executing the code.

let filesize s =
let tostr f =
  if f>9.9 then
    string_of_int (int_of_float (f +. 0.5))
  else
    let res = string_of_float (floor (f *. 10.0 +. 0.5) /. 10.0) in
    if String.length res=2 then
      res ^ "0"
    else 
      res
in
let bytes = Int64.to_float s in
if bytes > 512.0 then
  let kb = bytes /. 1024.0 in
  if kb > 512.0 then
    let mb = kb /. 1024.0 in
    if mb > 512.0 then
      let gb = mb /. 1024.0 in
      tostr gb ^ " Gb"
    else
      tostr mb ^ " Mb"
  else
    tostr kb ^ " kb"
else
  Int64.to_string s
;;

The ocaml standard library has a set of conversion functions. These functions normally follow the form of 'int_of_float' and 'string_of_float'. Specific types like 'Int64' use shorthand notations like 'Int64.to_float'. String concatenations are done with the operation '(^)'. Normally, functions are defined for only one specific type, so there are new sets of arithmetic functions for floats like '(+.)', '(*.)' and '(/.)'. The 'tostr' sub-function has some extra calculation to change something like '5. Gb' into the nicer form of '5.0 Gb'.

List notation and type conversion

The next function, 'converttime', converts a string into a float. OCaml uses floats for date for 2 reasons. The first is to prevent possible Year 2k problems, and can also be used for less than one-second time measurements. The function accepts English acronyms for month names. So let's introduce the list and the pair to create a translation of acronyms into numbers.

let month = [("jan", 0); ("feb", 1); ("mar", 2); ("apr", 3); ("may", 4); ("jun", 5);
             ("jul", 6); ("aug", 7); ("sep", 8); ("oct", 9); ("nov", 10); ("dec", 11)]
;;

This list is totally static, and can be used easily by the standard function List.assoc to convert a string into the corresponding number.

let converttime str =
try
begin match
  if str>"a" && str<"z" then
  ( int_of_string (string_sub str (String.rindex str ' '+1) 99),
    List.assoc (string_sub str 0 3) month,
    1
  )
  else
  ( int_of_string (string_sub str 0 (
      try String.index str '-' with Not_found -> 99
    )),
    ( try let pos=String.index str '-'+1 in
        int_of_string (string_sub str pos (
          try String.index_from str pos '-'-pos with err -> 99
        ))-1
        with err -> 0
    ),
    ( try let pos=String.index str '-'+1 in
        int_of_string (string_sub str (String.index_from str pos '-'+1) 99)
        with err -> 1
    )
  )
with (yr,mn,md) ->
(* print_string ("Last access before: "^
     string_of_int (if yr<50 then yr+2000 else if yr<100 then yr+1900 else yr)^"-"^
     string_of_int (mn+1)^"-"^
     string_of_int md^"\n"); 
*)
  fst (mktime 
  { tm_sec = 0; tm_min = 0; tm_hour = 0;
    tm_mday = md; tm_mon = mn;
    tm_year = if yr<50 then yr+100 else if yr<100 then yr else yr-1900;
    tm_wday = 0; tm_yday = 0; tm_isdst = false
  })
end with err ->
  print_string ("Cannot decipher this date string '" ^ str ^ "'\n"); max_float
;;

The new operation in this function is the 'match <expr> with <template> -> expr'. This is one of the most versatile instructions of ocaml. It can be used to examine the contents of variables and get the needed information out of it. This function creates the triplet (year, month, day-of-month) out of 2 different date notations. To debug this function the 'print_string' instruction is included but commented out to prevent clutter in the output of the program. Normally there is some logging mechanism to make the extra messages optional for the user. The 'print_string' shows the ISO notation of the given date; it creates a 4-digits year and gives a month number with January=1 instead of the internal Unix use of January=0.

This function also shows the use of 'try <expr> with err -> <expr>' that caches every possible exception and fills the variable 'err' with the details of the exception. This function can raise quite a lot of different exceptions, and frankly I am not very interested in the details. The routine just complains to the user about the given date string and gets over it. It returns the maximal possible float to include every filename.

The main standard function is the 'Unix.mktime' function. It wants to get a record filled with numbers about the current time. This function returns a pair with the needed float and a normalized record. With the pervasives function fst returns just the first parameter of the pair.

The ';' before the 'max_float' indicates that the expression results in a float, but the instructions before the ';' are calculated first. This is the first non-functional instruction inside the example code. OCaml is not strictly functional, but has the full power of other functional languages.

Dynamic data structure

Now is the time for a real data structure that is dynamically build and can be used in a lot of different ways.

type entrytype =
| Dir of entry list   (* directory with a list of files *)
| File of string      (* a file inside a directory *)

and

entry = {
	mutable e_name: string;   (* name of a file or directory *)
	e_type: entrytype;        (* what type is this together with type
                                     related information *)
	e_atime: float;           (* last access time *)
	e_size: int64;            (* size of the file or size of all the matching
                                     files in the directory *)
}

The 'and' statement is used to glue the two definitions together. They are created at the same time so that 'entrytype' can include 'entry' and vice-versa. 'entrytype' can consist one of 2 things: a directory with a list of entries or a file with its type. The directory entry has a mutable name. This is can be used later on to change a filename info the full path to that file.

As with ANSI C, the operators for Boolean algebra are '(&&)' and '(||)'.

Recursion

let rec dirwrite el depth sortfn =
List.iter (
  fun e ->
    match e.e_type with
    | Dir lst ->
       if e.e_size <> Int64.of_int 0 then begin
         print_string ((String.make (depth*2) ' ') ^ "Directory " ^
           e.e_name ^ " = (" ^ filesize e.e_size ^ ")\n"); 
         dirwrite lst (depth+1) sortfn
       end
    | File string -> 
       print_string ((String.make (depth*2) ' ') ^ e.e_name ^
         " (" ^ filesize e.e_size ^ ")\n")
  ) (List.sort sortfn el)
;;

Here is the recursive ('rec') function 'dirwrite' that traverses a given tree 'el' and writes the result to the standard output. The parameter 'depth' indicates the amount of spaces to write a tree like structure of filenames. The function sorts all the lists with the given function 'sortfn'. The new language structure here is 'fun <parm-1> ... <parm-n> -> <expr>'. This construction creates a function without a name. The parameters of this function like construction can be used like a template to match pairs.

This function suppresses directories that are 0 bytes in size to reduce clutter.

Variables vs. definitions

(* List of global variables *)

let min_size = ref (Int64.of_int 0) and    (* minimum size of a file in bytes *)
    last_access = ref max_float and        (* last access time in seconds since 1970 *)
    has_type = ref "" and                  (* type of file to show or empty to
                                              show all *)
    name_match = ref "" and                (* regular expression to match the filename
                                              with; empty is show all *)
    name_regexp = ref (Str.regexp "") and  (* pre-calculated regular expression *)
    no_symlinks = ref false                (* don't follow symbolic links to
                                              directories *)
;;

This is a list of variables that can be changed due to the 'ref <expr>' construction. Normally definitions are just a label to their contents. These definitions are pointers to the memory and can be read by '!<variable>' and written by '<variable> := <expr>'. The parameters given to the program can make changes to the way the files are read.

let rec dirread path =
let list = ref [] and
    size = ref (Int64.of_int 0) in
try
let dh = opendir path in
while true do
  let file = readdir dh in
  if file<>".." && file<>"." && file<>"CVS" && String.sub file 0 1 <> "." then
  let s=stat (path^"/"^file) in
  if s.st_kind = S_DIR &&
    (not !no_symlinks || (lstat (path^"/"^file)).st_kind <> S_LNK)
  then
    let dir = dirread (path^"/"^file) in
    list := 
    { e_name = file;
      e_type = Dir (fst dir);
      e_atime = s.st_atime;
      e_size = snd dir
    } :: !list;
    size := Int64.add !size (snd dir)
  else if 
    (!has_type = "" || gettype file = !has_type) &&
    s.st_size > !min_size && 
    s.st_atime < !last_access &&
    (!name_match = "" || Str.string_match !name_regexp file 0)
  then begin
    list := 
    { e_name = file;
      e_type = File (gettype file);
      e_atime = s.st_atime;
      e_size = s.st_size;
    } :: !list;
    size := Int64.add !size s.st_size
  end
done;
(!list, !size)
with 
| End_of_file -> (!list, !size)
| Unix_error (EACCES, err, parm) -> (!list, !size)
;;

The following functions are introduced in the function 'dirread':

  • Unix.opendir to start reading a directory.
  • Unix.readdir to read a filename.
  • Unix.stat for a record (Unix.stats) of statistics on a file.
  • Unix.lstat for statistics on a link.
  • Int64.add to add two int64 type of variables
  • Str.regexp to create a new interpreted regular expression
  • Str.string_match to match a string against a regular expression
  • Pervasives.(::) to create a list with an extra element in front of the old one
  • Pervasives.true as a Boolean constant
  • Pervasives.snd to return the second part of a pair
  • exception Unix.Unix_error (EACCESS, err, parm) that is raised when an access denied is encountered.
  • There is also a new construction 'while <boolean-expr> do <code> done' it just does what it is supposed to do.

    Small is beautiful

    let rec flat el path =
    List.fold_right (
      fun e ls ->
        match e.e_type with
        | Dir lst -> flat lst (path ^ "/" ^ e.e_name) @ ls
        | File string ->
            e.e_name <- (path ^ "/" ^ e.e_name);
            e :: ls
      ) el []
    ;;
    

    This neat little routine 'flat' hits the tree 'el' flat on the ground. It takes every file from every branch and creates a single list of all the encountered files. This is done with one of the most versatile standard routines inside ocaml: the 'List.fold_right' routine. This routine introduces a state machine (scarab) that crawls over a list and operates on every encountered element. It produces a new structure (droppings) as a result -- in this case, a flattened list.

    The construction '<record-field> <- <expr>' changes the contents of a mutable record field. Without mutable fields, you can mutate records only by creating a new one with lots of fields inherited from the old one. This is a shortcut for that.

    let name_order a b =
    compare a.e_name b.e_name
    ;;
    
    let type_order a b =
    let typea = match a.e_type with Dir ls -> "dir" | File tp -> tp and
        typeb = match b.e_type with Dir ls -> "dir" | File tp -> tp in
    if compare typea typeb = 0 then
      compare a.e_name b.e_name
    else compare typea typeb
    ;;
    
    let atime_order a b =
    compare a.e_atime b.e_atime
    ;;
    

    A set of sorting functions to use inside 'dirwrite'. The function 'compare' results in the widely used values of -1 for lower than, 0 for equal and +1 for higher than.

    Command line parameters

    let dir = ref "." and
        sort = ref name_order and
        show = ref 0
        in
    
    Arg.parse [
       ("-t",Arg.Unit (fun () -> sort := type_order), 
         "Sort by type and filename");
       ("-l",Arg.Unit (fun () -> sort := atime_order),
         "Sort by last access time");
       ("-n",Arg.Unit (fun () -> show := 1),
         "List filenames");
       ("-b",Arg.Unit (fun () -> show := 2),
         "List both filenames and sizes");
       ("-s",Arg.Unit (fun () -> no_symlinks := true),
         "Don't follow symbolic links");
       ("--before",Arg.String (fun s -> last_access := converttime s),
         "Last access older than give date (format 'yyyy-mm-dd' or 'mmm yyyy')");
       ("--size",Arg.Int (fun i ->
            min_size := Int64.mul (Int64.of_int i) (Int64.of_int (1024*1024))
         ), "File size bigger than size in Mbytes");
       ("--type",Arg.String (fun s -> has_type := s),
         "File is specific type");
       ("--name",Arg.String (fun s ->
            name_match := s; name_regexp := Str.regexp (s ^ "$")
         ), "Filename matches regular expression")
    ] (fun d -> dir := d) "show [DIR]";
    let res = dirread !dir in
    if !show=0 then begin
      dirwrite (fst res) 0 !sort;
      print_string ("Total size " ^ filesize (snd res) ^ "\n")
    end else
      List.iter
        (fun e -> 
          print_endline (e.e_name ^ if !show=2 then " ("^filesize e.e_size^")" else "")
        ) (List.sort !sort (flat (fst res) !dir))
    ;;
    

    And here is the main routine. It calls the Arg.parse routine to parse the parameters given to the program. But this is too much un-GNU for me. I wrote my own version of it that follows the GNU coding standards a bit more than the default one (Gnuarg). The other version is a bit more complicated so I will include only the sources that use it.

    Generating binaries

    The code can be obtained from here. Just unpack it somewhere with 'tar -xzf show.tar.gz' and move into the source directory with 'cd show/src'. There is also a Makefile that compiles to machine code and installs everything. But Makefiles are too rough for sour eyes to show in this article. The nitty-gritty details are there in the source. The general compile form is.

    ocamlopt -o show unix.cmxa str.cmxa basics.cmx show.ml
    

    The only non-standard libraries in use here are unix.cmxa and str.cmxa.

    make
    su
    make install
    exit
    show --help
    show -s ~ --size 3 --before "apr 2003"
    

    That concludes this example program.

    Language features

    Garbage collector
    Just forget variables that contain complete data structures. Once it gets out of scope, the total structure will be eliminated from memory in due time.
    Flexible data-structures
    Any 2 data structures can be combined without hassle. Just create an array of records that contain 2 fields with hash tables of strings. No problem there... everything in a single variable than can be passed to functions or can be used globally in the program.
    No pointers needed
    A variable can have any type and when a new variable is created
    Flexible in language boundary checks
    The language can check array and string boundaries automatically, or those checks can be turned off for an extra speed boost. Without it, the language can give a segmentation fault, but that is the programmer's choice.
    High quality error handling
    Totally integrated into the language and no notable performance hit.
    Native code generator and byte code interpreter
    All the tools are there -- interpreter (ocaml), byte code (ocamlc) and native code compiler (ocamlopt) -- every wish is granted. The package comes also with a documentation generator (ocamldoc) and a simple to use profiler (ocamlprof) that adds usage counts as comments to the original source code. The language is also compatible with the more sophisticated profilers around.
    ANSI-C compatibility layer
    It is possible to include ANSI C routines inside OCaml programs, and OCaml routines inside C programs. This has a very easy to use API. Slightly less easy is the creation of OCaml data structures inside C; for me, that was the source of many segfaults. So, my routines call exported OCaml routines to fill data structures and create only OCaml strings and numbers in C. That way I won't have the hassle to debug the C code... OCaml is much easier to debug for me.
    Object orientation
    Not my favourite programming paradigm, but it is possible to build object-oriented programs in this language. Those features are not part of this article. I can live without them.
    An active mailing list
    This list is at caml-list@inria.fr and is normally in English. Yes, this originally French project has taken the burden to translate almost everything they got. This is no easy feat for them, so be grateful.

    Cons:

    Duplicate efforts in libraries
    There are separate libraries for different type of big arrays, big files, and extra long integers. This isn't a big problem, because you can always just start with the normal structures and drop in the special library when need arises. The naming of the different functions is very much standardized, so renaming of function calls isn't needed much. The extra long integers though are too much different from normal integers. That part of the standard functions really need some tuning.
    Readability
    You need to be familiar with the basis constructions of the language, to make any sense of the actual code. Some constructions can look really weird without intimate knowledge of the language. OCaml is not a very natural language and has a very powerful, short notation for things. But this not much worse than languages like ANSI C, Perl, or lisp.
    Not known enough in the Linux world
    This language has excellent interfaces to standard libraries and easy binding to ANSI C, but still isn't very known. I like to create some articles like this to change that a bit. This is a really great language to program in, and gives you real power without the pitfalls common in other languages. Programmers should give it a try and feel that power once.

    [BIO] Developer at a small technology firm in the Netherlands called V&S bv. (www.v-s.nl) We sell firewall, anti-virus and spam boxes based on the Linux OS. Using more and more the OCaml language to write my applications. Busy writing a lightweight http server with an internal scripting language (camlserv.sourceforge.net, source code here) Interested in writing AI based computer games. Always trying writing one, nothing ready yet.

    Copyright © 2004, Jurjen Stellingwerff. Copying license http://linuxgazette.net/copying.html

    Published in Issue 99 of Linux Gazette, February 2004