In Praise of Perl

From: Ron Pero 
To: Perl-Win32-Web Mailing List 
Subject: Re: Diversification? (career-oriented)
Date: Thursday, February 11, 1999 3:12 AM

1. There are some perl worksite success stories at the home page of
http://www.perl.com.

2. On the DBI (perl's cross-db database interface) email list, someone once
asked for success stories about the use of perl with databases at the
worksite, and received these excellent responses.

Ron
---
Here is an edited summary of responses to my call for success stories.
Lisa W. Nyman     Standard Disclaimer applies.
begin summary ---------------------------------------------------------

I'm with the CA Dept of Water Resources and we have a real-time hydrologic
database that's used for flood operations and forecasting.  Our database
is about 15 GB in size and growing at approximately 3 GB/year.  During the
last winter season (97-98), we had almost 782,000 hits to our system.  
Our system collects real-time data over satellite and microwave
transmission and is capable of generating real-time graphs of the data. We
are currently using DBI with DBD-Oracle for Oracle 8.0.3.  Our web
applications, of course, are written in Perl.  Our graphing applications
and data exchange applications are written in C.
If you'd like to try out our system, it's at http://cdec.water.ca.gov

-------

You might want to contact DejaNews (www.dejanews.com).  They are using
mod_perl with DBI and a multi-terrabyte database which includes
practically everything ever posted on USENET.  Its not slow, either.

-------

About two years ago, I built a system for NASA's Langley Research Center
in Virginia that puts a searchable Web front end on a database of about
100,000 NASA-owned equipment items.  I used Apache, DBI, Informix, WDB,
and mod_perl on a Sparc 20.  Ran like a charm.  They liked it so much they
used to give demos at meetings on reorganizing the wind tunnels. :) Thing
was, every time they showed it to people, I ended up extending the system
to add something new, like tracking equipment that was in for repairs, or
displaying GIFs of technical equipment so when they lost the spec sheet,
they could look it up online.  When it works, success feeds on itself. 

I started the project back in the infancy of Perl5, DBI, DBD::Informix,
and mod_perl.  I know I gave Mr. Leffler and Mr. MacEachern headaches
trying to figure out problems with me.  :)  However, the dedication and
instant response time of all of the Perl software developers cured many
show-stopper problems in time for me to make all my deadlines.  I can't
say enough about the quality of the software, the support, and the
developers who make this software available.  I don't even want to think
about trying to get a correct answer out of a major vendor in less than an
hour, but it happened many times with the Perl software developers.

-------

A year ago, I built a large data processing farm/website backend
for a market data firm using Informix, Perl DBI/DBD-Informix, etc. and
it's still in use today.  I can't name the firm, but you see their
research data on all the major sites.

-------

Motorola Land Mobile Products, on a global basis, is using perl with DBI
and DBD-Oracle as a standard web based reporting for significant portions
of the manufacturing and distribution part of the business. From the
manuacturing perspective there is a conversion away from the standard
Oracle Forms and SAP GUI reports to a pure web based platform (perl DBI
and DBD) for all reporting. The Engineering areas have also significant
investments in similar types of platforms. Several "good" sized
applications are in use, ranging from simple create a notification,
approve notification and mail notification to dynamic routing of approvals
to significant business applications. The perl platform has been more
stable than the Oracle development platform. While you need a bit more
"patience" to develop the web based applications (for user layouts that
look good) my experience is that time to implementation for the Oracle
Tools is slightly longer. "Repair" cycles are longer for the Oracle Tool
approach. The software quality of the perl approach has been better, but
that may be due to differences in software development methodology.

-------

Currently I am creating a virtual ISP site using perl/dbi/dbd::oracle. It
is a combo intranet/extranet/corporate site that will be used by all of
our client base. All work that I have done on our intranet is
perl/dbi/dbd::oracle.  This includes time_sheet tools, department request
tracking, conference room calendars, hr admin, and the list goes on and
on. I have been using DBI/DBD for at least two years and haven't had any
serious problems (most are oracle related and/or programmer error!!)

-------

British Telecom, Call Management Information System:

Collects ~100MB of call data per day from over 800,000 monitored
phone numbers. ~178GB processed so far (over 2.6 billion calls).
Data processed and loaded into Oracle using DBI and DBD::Oracle.
Database holds rolling data for around 19 million calls.
Generates over 25,000 PostScript very high quality reports per month
(~5 pages with 11 color graphs and 5 tables) generated by using perl
to manipulate FrameMaker templates.

The whole system runs on three dual processor Sun SPARC Ultra 2
machines. One for data acquisition and processing, one for Oracle
and the third does most of the report production (which is also
distributed across the other two machines). Almost the entire system is
implemented in perl.  

There is only one non-perl program and that's only
because it existed already and isn't specific to this system.  The other
non-perl code is a few small libraries linked into perl using the XS
interface. 

A quote from project summary by a senior manager: "Less than a
year later in August 1996 BT's CMI reports service went live for customers
on System X exchanges. This was subsequently celebrated as one of the
fastest projects of its size and complexity to go from conception to
launch and it's already 'defending' UKP 200 million of PSTN revenues."

Designed, developed, implemented, installed and supported by the Paul
Ingram Group, who received a "Rising to the Challenge" award from BT for
their part in the project. Without Perl, the system could not have been
developed fast enough to defend BT's major accounts from defecting to a
rival supplier. And without Perl, the system could not be so easily
maintained or so quickly extended to meet changing requirements.

-------

I don't have a fully implemented system of the scale which has already
been posted to the DBI mailing list (Tim's BT example is amazing), but I
am working on a system now which I think is interesting. It is implemented
using Perl, DBI, Apache (mod_perl), hosted using RedHat Linux 5.1 and
using a light-weight SQL RDBMS called MySQL.  The system is for a major
multinational holding company which owns approximately 50 other companies.  
They have 30,000 employees world-wide, who needed a secure system for
getting to web-based resources.  This first iteration of the Intranet is
specified to handle up to 40 requests for web objects per second
(approximately 200 concurrent users), and runs on a single processor Intel
Pentium-Pro with 512 megs of RAM. We develop in Perl using Object-Oriented
techniques everywhere.  Over the past couple years we have developed a
large reusable library of Perl code.  One of our most useful modules
builds an Object-Relational wrapper around DBI to allow our application
developers to talk to the database using O-O methods to access or change
properties of the record.  We have saved countless hours and $$$ by
building on Perl instead of a more proprietary system.

------- 

I certainly don't move anything like the volume of data that Tim
does, and I never speak for the Mouse, but he has a 'household name'. We
use it primarily for intranet web applications that involve generating
reports from SAP and custom systems like time tracking. There is also a
system that provides a security wrapper around sensitive content on our
intranet (more featured and flexible than basic authentication). I've also
used it to gather performance data when tuning systems. We've replaced
some shell/sql*plus stuff with DBI and gained performance and reliability.

------ 

Motorola is using perl / DBI for 80% of there factory control system
for Paging Systems Manufacturing (the equipment that sends pages to the
pager).  The system has been deployed in Texas and China and soon will be
deployed in Illinios.  Perl is used for test system interfaces as well as
user interfaces, all which communicate back to Oracle 8.

------------------------------------------------------------------------------

Giant Genomes
this was clipped from the perl-xml mailing list

Hi.
Some Context:
=============
We are beginning a collaboration with the D.O.E. Oak Ridge lab (ORNL)
on human genome databases.  The decision has been made within this group
(at D.O.E.) to standardize on XML as an interchange format (and perhaps
more).  These biological databases are getting very large (Genbank is
perhaps 12 GB) and have been growing rapidly.  The human genome project
has now sequenced about 10% of the human genome over the past few years
and is accelerating --- some groups expect to finish by the end of
next year.  That would imply that the sequence databases could be about
100 GB by Dec 2000.
We have recently obtained a 64 processor Origin 2000 with 16GB of RAM and 
a 600 GB fibre-channel raid array (this was not my idea ...).  I only
mention this to indicate that we have the facilities to look at *really
large* "documents."
Scenario:
========
The genbank format would be trivial to convert to XML with perl (I could
write the script in a few minutes).  Essentially, I could turn the entire
genbank into a single XML document.  I don't think that this is necessarily
a good idea but I'd like to get some idea of how current XML technology
would handle this.
I've been following XML development (and web in general) quite closely for
some time (I know of Tim Bray, having worked for Open Text).  Intuitively, 
I have the idea that the XML Specification places constraints on an XML parser
which make it logical to try to read the entire document into memory (I've
seen an example where this is done in perl:
     undef($/);
     $file=<>;
["XML: Can the Desperate Perl Hacker Do It ?" by Michael Leventhal] ).
Questions:   [some of these are sort-of rhetorical]
==========
What would be the impact on an XML parser trying to read a 12GB document ?
Does it have to read the entire thing into memory (we do have enough at
the moment :) ?
Does expat do this ?
Does anyone have any idea what it would be like to try to use Perl-Xml
for this kind of problem (performance etc) ?
What happens when our document is 10 times the size of RAM as it will be
in 18 months or so ?
----
Guy Hulbert, Systems Manager Bioinformatics Supercomputing Centre



XML - Big Money
this was clipped from the perl-xml mailing list

> if Tim Bray, who may reasonably be expected to have
> some clues concerning XML, can't get XML->HTML transformation going with
> existing XSL yet, reliably, I think I'll conclude that XML (important
> chunks of it) isn't close to being ready for prime time yet.

In a browser context that may be true. But XML is much, much bigger
than that. I just wrote a case study on a new Dun and Bradstreet
toolkit/platform called Global Access (globalaccess.dnb.com). In
brief: they're normalizing their whole worldwide inventory of
financial data on millions of companies, using two sets of
XML-governed interfaces. On the back end, a hodge-podge of mainframes,
"legacy" client-server apps, and other stuff gets pumped into XML in a
data-tier aggregation service. Then a middle tier, based on
webMethods' XML-oriented B2B Integration Server, presents common
services -- authentication, data-feed customization and fetching, all
defined in XML -- to a client toolkit implemented in both COM and
ActiveX.

The upshot? Here's the best example I came up with: a D&B customer --
a credit insurer -- built an application that writes policies for its
customers. The credit insurer's app talks XML to the D&B middle tier,
which in turn talks XML to the D&B data tier. Where before it was
months for the credit insurer to get D&B to develop custom feeds to
support the risk analysis needed to write these policies, now the
credit insurer in days can reach directly into the D&B data inventory
and do the customization on its own. Hot stuff indeed! DCOM?
IIOP/CORBA? Nah. XML-RPC. That's what's going to make the object Web
real, and right quick too.

Interestingly, the D&B toolkit wraps the XML up in an object model
that you twiddle in Java or VB/COM, so nary an angle bracket or parser
is visible to the developer. But...they're very clear as to the
importance of having the entire system, end to end, be equally usable
in "raw" mode by just shooting XML over HTTPS connections. They don't
want to lock customers into their toolkit, they're not even selling
the toolkit, they're selling the data and they want to construct
services that people get at any which way. They picked XML as the
enabling technology and really went to town with it.

So my point is, you can't just XML's ready-for-primetimeness solely,
or even primarily, by the state of its implementation in browsers.
It's busily taking the middleware market by storm. Pick up a copy of
Computerworld, all you read about is SAP, PeopleSoft, ERP (enterprise
resource planning) rollouts, supply-chain management. All this stuff
is going XML, bigtime, and this is the kind of stuff that makes the
world turn.
 Jon Udell | http://udell.roninhouse.com/ | 603-355-8980
Jon Udell was formerly an editor for Byte magazine.