Rev 7 | Blame | Compare with Previous | Last modification | View Log | RSS feed
<?xml version="1.0" standalone="no"?>
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<!--
$Id$
-->
<book>
<bookinfo>
<title>Validation of TenDRA Capability to Implement a Set of Commands for
the Linux Operating System</title>
<corpauthor>The TenDRA Project</corpauthor>
<authorgroup>
<author>
<firstname>François</firstname>
<surname>de Ferrière</surname>
<affiliation>
<orgname>Open Software Foundation Research Institute</orgname>
</affiliation>
</author>
<author>
<firstname>Fred</firstname>
<surname>Roy</surname>
<affiliation>
<orgname>Open Software Foundation Research Institute</orgname>
</affiliation>
</author>
<editor>
<firstname>Jeroen</firstname>
<surname>Ruigrok van der Werven</surname>
</editor>
</authorgroup>
<abstract>
<para>This report describes work done under contract to the Defence
Research Agency (DRA) of the U.K. It is an extension of an earlier
contract to assess the capability of the DRA TenDRA technology to
express a fully portable operating system implementation.</para>
</abstract>
</bookinfo>
<chapter>
<title>Executive Summary</title>
<para>During a first phase of the project the main goal was to examine
the extent to which the TenDRA technology could be used to compile a
complete operating sytem. This experiment was carried out on the
Unixware operating system on an Intel 486 platform. Although Unix
sources are known to be compiler dependent, it was found that most of
the code could be compiled with no, or minor, modifications. Details
of the results can be found in a
<biblioref linkend="Ferrière95-2">report</biblioref>.
</para>
<para>The goal of this second phase of the project was
to study the feasibility of producing Unix commands in architecture
neutral (ANDF) format which could be readily ported to different
hardware platforms running the same operating system. The target
system chosen for this second phase was Linux, which was available on
both the Intel and DEC Alpha platforms (although the latter was
incomplete at the beginning of the project).</para>
<para>The experiment carried out during this second phase was very
successful. A significant amount of complex code was converted to the
architecture neutral ANDF format, and the portability of this code was
demonstrated. However, due to time constraints, the number of commands
ported to both platforms was more limited than had been hoped. The
project also provided some interesting lessons about the strengths and
limitations of the ANDF/TenDRA technology and about API issues.This is
the final report on work undertaken on the Linux operating system on
the Intel/i386 and Digital/Alpha platforms.</para>
</chapter>
<chapter>
<title>2. Objectives and Description</title>
<sect1>
<title>2.1 Objectives</title>
<para>In previous work, we performed validation, performance and
robustness testing of the TenDRA technology to ensure its capability
to implement and fully bootstrap a UNIX-like operating system. We also
provided an assessment of the capability of TenDRA technology to
express a fully portable operating system implementation. This work
was very successful, and the results are reported in a
<biblioref linkend="Ferrière95-2">summary report</biblioref>.
</para>
<para>However, though we originally planned to conduct this first
experiment on two different architectures, an Intel/486 and a
Sun/Sparc platforms running UnixWare, it was completed only for the
Intel/486 platform. After discussions with DRA, it was then decided
to focus the second part of the project on the Unix commands, and to
switch from the UnixWare to the Linux operating system. The
motivations for a revised plan were:</para>
<itemizedlist>
<listitem>OSF is developing a Linux server for the Intel/486 and
PowerPC platforms, and we would like to deliver the set of
associated commands in ANDF format. In the event, native Linux
commands for PowerPC became available in the meantime.</listitem>
<listitem>Repeating on the Sparc platform the work already done on the
i486 platform would bring little added-value, while requiring a
significant amount of work. The major benefit of the work on a
second platform would be to demonstrate that a set of commands along
with its API can be defined in ANDF format and then installed on two
different platforms.</listitem>
</itemizedlist>
<para>Thus, the objective of the second part of the project is the
production in ANDF format of Linux commands, and their installation on
two platforms. This will demonstrate the ability of the TenDRA
technology to produce a set of architecture neutral commands and, at
completion of the project, will provide a set of freely distributable
commands in ANDF format.</para>
<para>The project, which lasted 9 months, started on July 1995, and was
finished at the end of March 1996.</para>
<para>This report summarizes all the work done under the contract for
this second part of the project.</para>
</sect1>
<sect1>
<title>2.2 General description</title>
<para>The commands, one part of a Unix system, are based on some
standard APIs, XPG3 for example, plus some extensions, which together
form the interface shared with the libraries against which they are
built. The commands should not have any assembly code, unlike the
other parts of a Unix system.</para>
<para>As for other software OSF already ported to ANDF, the port of the
Linux commands is done in three steps:</para>
<itemizedlist>
<listitem>The NAT-NAT step, which consists in rebuilding the commands
with the native compilation chain, to ensure that they can be
regenerated from their source files.</listitem>
<listitem>The DRA-NAT step, for which the TenDRA technology is used as
a replacement of the native compilation chain to build the commands,
using the native system header files, as for a classical compilation
chain. This part involves dealing with discrepancies between the
native and the TenDRA code generators.</listitem>
<listitem>The DRA-DRA step, which will consist in using the TenDRA
technology as a portability tool. The API shared by the commands and
libraries is defined, and used to produce the commands in
architecture independent ANDF code. This code will be installed and
validated on the selected machines.</listitem>
</itemizedlist>
<para>We initially planned to conduct the experiment on the Intel/i386
and IBM/PowerPc platforms, both running the Linux operating system.
However, the Linux system for the IBM/PowerPC platform was still under
development at the time we needed it, in December 1995. So we decided
to replace it by a Digital/Alpha platform, the only other platform for
which a Linux port was sufficiently advanced at that time. However,
it is a 64-bit platform, so this switch was more of a challenge,
because of the fundamental change in data sizes. It provided
additional tests of the TenDRA portability attributes.</para>
<para><A NAME="90"></A></para>
</sect1>
</chapter>
<chapter>
<title>3. Description of Project phases</title>
<para>In this section we specify the tasks which have been performed under
this project.</para>
<para>The set of commands which were ported to ANDF was split into two
subsets:</para>
<itemizedlist>
<listitem>The <I>level 1</I> subset corresponds to the level achieved
with UnixWare at the end of the first project (about 100 commands out
of 600). These commands conform to a standard interface (Posix or
XPG) with some simple extensions. They do not use extensions which are
difficult to port from one system to another. This set includes about
150 Linux commands.</listitem>
<listitem>The <I>level 2</I> subset corresponds to the maximum which can
be reasonably achieved for a given system. While we could expect about
twice as many commands as for level 1, we only compiled about 75 more
commands.</listitem>
</itemizedlist>
<sect1>
<title>Phase 1: level 1 commands on Intel/i386</title>
<para>The objective is to produce a ``level 1'' set of Linux commands in
ANDF format for the Intel platform. This requires the production of
the associated API, also in ANDF format (token library).</para>
<para>The major tasks for this phase are:</para>
<sect2>
<title>T1. Linux installation.</title>
<itemizedlist>
<listitem>Install the Linux system.</listitem>
<listitem>Install a compilation environment.</listitem>
<listitem>Install the Linux source code.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite:</emphasis> Linux system for
Intel.</para>
<para><emphasis>Delivery:</emphasis> System running.</para>
</sect2>
<sect2>
<title>T2. TenDRA installation.</title>
<itemizedlist>
<listitem>Install the TenDRA technology for Linux.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite:</emphasis>TenDRA technology &
T1.</para>
<para><emphasis>Delivery:</emphasis> TenDRA installed.</para>
</sect2>
<sect2>
<title>T3. Level 1 commands port.</title>
<itemizedlist>
<listitem>Define the level 1 set of Linux commands.</listitem>
<listitem>Compile the level 1 commands in NAT-NAT mode.</listitem>
<listitem>Compile the level 1 commands in DRA-NAT mode.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite:</emphasis> Linux source code, T1 &
T2.</para>
<para><emphasis>Delivery:</emphasis> Level 1 commands compiled with
TenDRA in native mode.</para>
</sect2>
<sect2>
<title>T4. Level 1 commands API definition.</title>
<itemizedlist>
<listitem>Define the non-explicit API used by this set of commands.
Machine dependent code issues will be addressed
specifically.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite:</emphasis> Linux source code.</para>
<para><emphasis>Delivery:</emphasis> Set of ANDF header files for this
level 1 API.</para>
</sect2>
<sect2>
<title>T5. ANDFization of Level 1 commands.</title>
<itemizedlist>
<listitem>Produce the level 1 commands with the TenDRA technology,
using the ANDF definition of the API defined in the previous
task.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite</emphasis>: Linux source code, T2 &
T4.</para>
<para><emphasis>Delivery</emphasis>: Level 1 commands in ANDF
format.</para>
</sect2>
<sect2>
<title>T6. Level 1 commands API installation.</title>
<itemizedlist>
<listitem>Build the token library for the level 1 commands
API.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite</emphasis>: T2 & T4.</para>
<para><emphasis>Delivery</emphasis>: Token library for this level 1
API.</para>
</sect2>
<sect2>
<title>T7. Level 1 commands installation and validation.</title>
<itemizedlist>
<listitem>Install the commands in ANDF format produced in task T5.
</listitem>
<listitem>Validate the commands using adhoc tests.</listitem>
<listitem>Write a report that describes the results obtained and
the problems encountered.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite</emphasis>: T2, T5 & T6.</para>
<para><emphasis>Delivery</emphasis>: Report on level 1 commands on
Intel.</para>
</sect2>
</sect1>
<sect1>
<title>Phase 2: level 1 commands on Digital/Alpha</title>
<para>The objective is to validate that the commands produced during the
first phase can be easily ported to the Alpha platform.</para>
<para>The major tasks, during this phase, are:</para>
<sect2>
<title>T8. Linux installation.</title>
<itemizedlist>
<listitem>Install the Linux system.</listitem>
<listitem>Install a compilation environment, including header files
and libraries.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite</emphasis>: Linux system for
Alpha.</para>
<para><emphasis>Delivery</emphasis>: System running.</para>
</sect2>
<sect2>
<title>T9. TenDRA installation.</title>
<itemizedlist>
<listitem>Adapt the TenDRA technology for Linux on the
Alpha.</listitem>
<listitem>Install the TenDRA technology.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite</emphasis>: TenDRA technology &
T8.</para>
<para><emphasis>Delivery</emphasis>: TenDRA installed.</para>
</sect2>
<sect2>
<title> T10. Level 1 commands API installation.</title>
<itemizedlist>
<listitem>Build the token library for the level 1 commands
API.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite</emphasis>: T4 & T9.</para>
<para><emphasis>Delivery</emphasis>: Token library for this level 1
API.</para>
</sect2>
<sect2>
<title> T11. Level 1 commands installation and validation.</title>
<itemizedlist>
<listitem>NAT-NAT and DRA-NAT check.</listitem>
<listitem>Install the commands in ANDF format produced in task
T5.</listitem>
<listitem>Validate the commands using adhoc tests.</listitem>
<listitem>Write a report that describes the results obtained and
the problems encountered.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite</emphasis>: T5, T9 & T10.</para>
<para><emphasis>Delivery</emphasis>: Report on level 1 commands on
Alpha.</para>
</sect2>
</sect1>
<sect1>
<title>Phase 3: level 2 commands on Intel/i386 and Digital/Alpha</title>
<para>The objective during this phase is to validate further the ANDF
tools by trying to extend the ANDF commands to a broader set that will
include some ``difficult'' cases.</para>
<para>The major tasks are:</para>
<sect2>
<title>T12. Level 2 commands port.</title>
<itemizedlist>
<listitem>Define the level 2 set of Linux commands, by extension of
the level 1 set.</listitem>
<listitem>Compile the level 2 commands in NAT-NAT mode on
Intel.</listitem>
<listitem>Compile the level 2 commands in NAT-NAT mode on
Alpha.</listitem>
<listitem>Compile the level 2 commands in DRA-NAT mode on
Intel.</listitem>
<listitem>Compile the level 2 commands in DRA-NAT mode on
Alpha.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite</emphasis>: Linux source code, T1, T2, T8
& T9.</para>
<para><emphasis>Delivery</emphasis>: Level 2 commands compiled with
TenDRA in native mode.</para>
</sect2>
<sect2>
<title>T13. Level 2 commands API definition.</title>
<itemizedlist>
<listitem>Extend the level 1 API to include the interfaces used by
the level 2 commands.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite</emphasis>: Linux source code &
T4.</para>
<para><emphasis>Delivery</emphasis>: Set of ANDF header files for this
level 2 API.</para>
</sect2>
<sect2>
<title>T14. Level 2 commands ANDFization.</title>
<itemizedlist>
<listitem>Produce the level 2 commands with the TenDRA technology,
using the ANDF definition of the API defined in the previous
task.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite</emphasis>: Linux source code, T2, T9
& T13.</para>
<para><emphasis>Delivery</emphasis>: Level 2 commands in ANDF
format.</para>
</sect2>
<sect2>
<title>T15. Level 2 commands API installation.</title>
<itemizedlist>
<listitem>Build the token library for the level 2 commands API on
Intel.</listitem>
<listitem>Build the token library for the level 2 commands API on
Alpha.</listitem>
</itemizedlist>
<para><emphasis>Prerequisite</emphasis>: T2, T9 & T13.</para>
<para><emphasis>Delivery</emphasis>: Token library for this level 2
API.</para>
</sect2>
<sect2>
<title>T16. Level 2 commands installation and validation.</title>
<itemizedlist>
<listitem>Install the commands in ANDF format produced in task T14
on Intel.</listitem>
<listitem>Install the commands in ANDF format produced in task T14
on Alpha.</listitem>
<listitem>Validate the commands using adhoc tests.</listitem>
<listitem>Write an intermediate report and a final
report.</listitem>
</itemizedlist>
<para><I>Prerequisite</I>: T2, T9, T14 & T15.</para>
<para><I>Delivery</I>: Intermediate report on level 2 commands, and
final report.</para>
</sect2>
</sect1>
</chapter>
<chapter>
<title>4. Project environment</title>
<sect1>
<title>4.1 LINUX operating system</title>
<para>Linux is a free Unix-like operating system, first developed by
Linus Torvalds on an Intel platform; ``official'' releases exist since
October 1991.</para>
<para>It has now been ported to Digital/Alpha and ports to other
machines, including PowerPC andPowerMAC, are under way.</para>
<sect2>
<title>Linux on Intel/i386</title>
<para><A NAME="10407">There are many distributions of Linux for the Intel
platform; we installed the Slackware distribution, based on the Linux 1.1
version.</A></para>
<para><A NAME="10408">We downloaded it from the ftp site:<BR/></A><A
HREF="ftp://sunsite.unc.edu/pub/Linux/distributions/slackware">sunsite.unc.edu:/pub/Linux/distributions/slackware</A></para>
<para><A NAME="10536">The version we installed was Linux 1.1.59, available since
October 1994. Since then, newer versions have been released, but we stuck to
this version throughout the project since it worked well, and because all the
packages it included were easily available in source form (see below).</A></para>
</sect2>
<sect2>
<title>Source code for Linux commands used by the project</title>
<para><A NAME="10411">We downloaded the source code of the Linux commands from
the ftp site:<BR/></A><A
HREF="ftp://sunsite.unc.edu/pub/Linux/distributions/slackware/source">sunsite.unc.edu:/pub/Linux/distributions/slackware/
source</A>.</para>
<para><A NAME="10397">This means that the versions of commands available on our
development machine for Linux/i386 were matching the source code we used as base
for the project, except in a few cases for which the source code had been
revised.</A></para>
</sect2>
<sect2>
<title>Linux on Digital/Alpha</title>
<para><A NAME="1551">For the Alpha platform, Linux is available from the `BLADE'
distribution, and more recently from the Red Hat distribution.</A></para>
<para>A 32-bit version of Linux/Alpha was first released in
January 1995; then a 64-bit version was available in November 1995, which
included most of the capabilities provided by the Linux/Intel system. We
downloaded the BLADE_0.3 release, consisting of more than thirty floppy images,
from the ftp site:<BR/><A HREF="ftp://ftp.digital.com/pub/DEC/Linux-Alpha">ftp.digital.com:/pub/DEC/Linux-Alpha</A></para>
<para>Since December 1995, another Linux/Alpha distribution has
become available from the RedHat company; it is built from the same components,
but newer versions, as the BLADE release.</para>
<para>An interesting feature of the current Linux/Alpha ports, is
that they provide rather extensive binary compatibility with Digital Unix. This
compatibility has been used to cross-build on Digital Unix for Linux/Alpha, and
also for a few features which were not available in the Linux/Alpha BLADE
release.</para>
</sect2>
</sect1>
<sect1>
<title>4.2 Hardware platforms and environment</title>
<para>The Intel platform was an Intel/i486 PC machine, with
most disk space available through NFS.</para>
<para>The Digital/Alpha platform was built specifically for
this project, around a <I>Digital AXPpci 33</I> motherboard. In fact, at the
time we set-up the machine, the Linux/Alpha ports were only running on a few
Alpha-based machines.</para>
<para>A Linux kernel had to be rebuilt for this machine in order
to add support for the 3COM Ethernet board we used, and for the NFS-client
capability. Most disk space was thus available through an NFS file system,
shared with the Intel platform.</para>
</sect1>
<sect1>
<title>4.3 TenDRA technology</title>
<para>We started the project on an Intel platform with a snapshot of the
TenDRA technology from April 1995, which included support for the
Linux/Intel platform. This snapshot was based on the ANDF 3.1
specification.</para>
<para>We switched to the November 1995 TenDRA snapshot, based on ANDF 4.0,
when we setup the second platform, in order to use the tools for the
Digital Unix/Alpha platform. Because of the high degree of compatibility
between Digital Unix and Linux on Alpha, we could use the TenDRA
technology on a DigitalUnix/Alpha platform to cross-build executables for
the Linux/Alpha platform.</para>
<para>The ANDF 4.0 intermediate file format is not upward compatible with
the ANDF 3.1 one, which required that we rebuild the intermediate ANDF
files for the Linux commands we had already built.</para>
<para>ANDF 4.0 contains increased capability, though not required by this
project, and forms the basis for the X/Open preliminary
specification XANDF.</para>
</sect1>
</chapter>
<chapter>
<title>5. Descriptions and Results of Project phases</title>
<para>In the next paragraphs, we describe the way we accomplished
the various tasks of the project and we summarize their results.</para>
<sect1>
<title>5.1 Linux installation</title>
<para>At the beginning of the project, we installed the Linux operating
system release 1.1 on an Intel/i386 machine. In December 1995, after a few
months of work on the Intel/i386 platform, we installed Linux on the
second platform for the project, which is a Dec/Alpha. Linux was first
released on this platform at the beginning of 1995.</para>
<sect2>
<title>Linux/i386 installation (including the source code for commands)</title>
<para>A machine with Linux 1.1.59 from the Slackware
distribution, including the native compilation chain and libraries from GNU, was
setup for the project.</para>
<para>The Linux system is available on several anonymous ftp
sites. The one we used was at sunsite.unc.edu, where a distribution of the
sources and binaries of the Intel/Linux commands from Slackware was available
under the /pub/Linux/distributions/slackware directory. Note that the current
Slackware distribution at the time of writing of this report is based on Linux
3.0.</para>
<para>In the Slackware Linux distribution for Intel/ix86, the
delivery of the source code for commands is split into a large number of
<I>packages</I>. The contents of each source package must be compiled
and installed individually. For example, the <I>awk</I> command,
actually <I>gawk</I>, belongs to the <B>bin</B> package which contains
56 commands, while the <I>bc</I> command belongs to the <B>bc</B>
package which contains only this one command. Consequently, we did not
download the whole set of sources for the Linux commands, but selected a
few packages containing the sources of the commands we intended to build
first. We also had a look at the Caldera Linux source distribution, and
it appeared to be organized in the same way.</para> <para>
A Slackware Linux package for source code distribution is
made of a compressed tar files (usually only one), optional patch
file(s), and a shell script. The execution of this shell script installs
the source files from the tar file(s), applies patches if necessary,
optionally performs a self-configuration step, runs the makefile(s) for
the compilation, and finally generates a binary package which holds the
resulting executables. This procedure has been adapted to fit the
NAT-NAT, DRA-NAT and DRA-DRA development steps on two platforms, as
described in the section<A HREF="linux_re.htm#4846">Setting up the
build environment</A>.</para>
<para>Note that each package has a private version number. Thus
packages can be maintained and released independently. Moreover, some packages
(e.g. the <B>bin</B> package) are a collection of several `subpackages', each
of which has its own version number.</para>
</sect2>
<sect2>
<title>Linux/Alpha installation</title>
<para>The Linux Operating System port to the Digital Alpha architecture
started two years ago. The first user-installable distribution was
available in January 95, from the `BLADE' distribution, and was a 32-bit
version. Then came a 64-bit version which was made compatible with Digital
Unix with respect to basic C language types:</para>
<informaltable>
<row>
<entry>int:</entry>
<entry>32-bit</entry>
</row>
<row>
<entry>long:</entry>
<entry>64-bit</entry>
</row>
<row>
<entry>pointer:</entry>
<entry>64-bit</entry>
</row>
</informaltable>
<para>While it is still under development, Linux/Alpha is now robust and
includes most of the capabilities provided by the Linux/Intel
system.</para>
<para>The BLADE distribution was the first available distribution for
Linux/Alpha. For the project, we retrieved the November 95 BLADE_0.3
release, based on the Linux 1.3 development kernel, at the following
site:</para>
<para><A HREF="ftp://ftp.digital.com/pub/DEC/Linux-Alpha"> ftp.digital.com:/pub/DEC/Linux-Alpha</A></para>
<para>This release consists of more than thirty 3.5'' floppy images (not
including X-Window). The source code for the commands is not a part of
this distribution. Since then, several new versions for the boot firmware,
kernel, compiler and libraries have been released, but, as we encountered
minor problems only with BLADE_0.3, we did not upgrade our system.</para>
<para>Since December 95, another distribution of Linux for Dec/Alpha is also
available from the RedHat company; the current version is:</para>
<para><A HREF="ftp://ftp.redhat.com/pub/redhat/redhat-2.1/axp">ftp.redhat.com:/pub/redhat/redhat-2.1/axp</A></para>
<para>This distribution includes all the source packages for the components
it is made from, along with some fixes and additions, in both binary and
source forms. It is possible to unpack a RedHat Linux/Alpha 2.1 set while
not running the RedHat Linux, but, as a proprietary packaging format is
used, one should install the packaging tools (<I>rpm</I>) first.</para>
<para>At the time we setup the machine, Linux/Alpha was operational only on
a few variants of Digital Alpha-based systems. So, we selected an entry
level and rather inexpensive board, the Digital AXPpci 33
Alpha PC motherboard, around which we built a machine. Our Linux/Alpha system
currently comprises the following:</para>
<itemizedlist>
<listitem>an 8-slot enclosure with 200W power supply and fan</listitem>
<listitem>Digital AXPpci 33 motherboard, Windows NT (ARC) firmware, PS/2
style keyboard interface, 233 Mhz Alpha processor</listitem>
<listitem>2x16MB, 36-bit, 70ns SIMMs</listitem>
<listitem>256 KB, 20ns cache [optional part]</listitem>
<listitem>a Number 9 GXE VGA display adapter (ISA)</listitem>
<listitem>a dumb VGA display</listitem>
<listitem>a PS/2 style keyboard</listitem>
<listitem>a 3.5''/1440K floppy disk drive</listitem>
<listitem>a SCSI-2 hard disk (a DECpc 2.0GB disk from Digital)</listitem>
<listitem>a 3COM Ethernet Link-II (aka 3c503) controller (ISA).</listitem>
</itemizedlist>
<para>We installed the BLADE_0.3 distribution on our machine, including the
C compilation chain and libraries. In order to use the 3COM Ethernet
board, we had to rebuild the kernel. We used almost all of the default
kernel build parameters, except for the Ethernet adapter, for the settings
for the `TGA graphics' support (switched to `no') and for the NFS-client
feature (selected). Note that a kernel rebuild takes more than half an
hour on our system.</para>
<para>A very interesting feature of the current releases of
Linux/Alpha is that they provide an almost perfect <B>binary compatibility
with Digital Unix</B>. This was of great help to us, as will be described
later.</para>
<para>Among the various updates to the Linux/Alpha boot
loader, kernel, C compiler and libraries, commands, ..., which have been
made by the Linux-Alpha development teams, we have used only a
few:</para>
<itemizedlist>
<listitem>upgrade of the <command>sed</command> command: some sed scripts
used for modifying the system headers when building the APIs with TenDRA
caused the original <command>sed</command> command to abort.</listitem>
<listitem>upgrade of a few system headers, extracted from the azstarnet
<filename>inc-and-libs-0.38.tar.gz</filename> file.</listitem>
</itemizedlist>
<para><A NAME="2048">These two updates were downloaded from the
site</A></para>
<para><A HREF="ftp://ftp.azstartnet.com/pub/linux/axp">
ftp.azstartnet.com:/pub/linux/axp</A></para>
<para><A NAME="6416">We encountered a few problems with the BLADE_0.3
release on the AXPpci Alpha board:</A></para>
<itemizedlist>
<listitem>The floppy disk driver sometimes entered a time-out, as
indicated by a console message.</listitem>
<listitem>Some shell scripts failed until a #!/bin/sh line, or equivalent,
was inserted. According to a member of the Linux/Alpha development team,
it is caused by the kernel command loader, and was fixed in new kernel
releases. We worked-around this problem by patching a number of shell
script files included with various source packages we were building on
Linux/Alpha: we realized too late that it would have been preferable to
upgrade the kernel.</listitem>
<listitem>Linux/Alpha failed to mount an NFS file system served by a HP-UX
release 8 machine. Fortunately, this problem disappeared when using a
server running Solaris or HP-UX release 9: we had to move our
development tree to such a host.</listitem>
<listitem>As mentioned above, the sets of source files were shared through
NFS between a Linux/Intel, a Digital Unix and a Linux/Alpha platform.
Occasionally, Linux/Alpha lost access to a file that had been updated
recently by another NFS client: the error message `<I>Stale NFS file
handle</I>' was displayed. Unmounting/remounting the NFS file system
usually cured such problems.</listitem>
<listitem>During kernel rebuilds, the compilation of at least one file
failed because of lack of memory: in the makefiles for kernel rebuild,
the gcc compiler is called with the `-pipe' option, which speeds up the
build but is not safe when compiling large source files. We wrote a
small shell script which redoes a compilation without the `-pipe'
option. (This problem was fixed by a subsequent kernel release:
<I>tcpip.c</I> was split into several parts...)</listitem>
</itemizedlist>
</sect2>
</sect1>
<sect1>
<title>5.2 TenDRA installation</title>
<para>We first installed the TenDRA technology on the Linux/i386
platform, from the April 1995 snapshot. Later, we installed the November 1995
release, the first to include support for the Dec/Alpha machine, in order to
start work on the Linux/Alpha platform. However, because this snapshot was not
upward compatible with the previous TenDRA release, we also had to install it on
the Linux/i386 platform. We did not upgrade to the February 1996 snapshot,
though it is compatible with the previous one. We only used it in a few cases
when we had a bug in a command and wanted to make sure that it was not due to a
problem already fixed in TenDRA.</para>
<sect2>
<title>TenDRA installation on Intel/i386</title>
<para>The TenDRA snapshot from April 1995, based on TDF 3.1,
included support for the Linux/i386 platform. So, the installation on our
machine was straightforward. We only had to recompile the tcc driver, and to
modify some environment and startup files to fine tune the level of checking.</para>
<para>When we started to work on the second platform, we had to
install the November 1995 snapshot of the TenDRA technology (see below). This
was DRA's first snapshot based on TDF 4.0, and it included significant changes
to the installation procedure. We had some difficulties to install this
snapshot, due to a few bugs in the new installation procedures, but once
installed the technology appeared to work well.</para>
<para><A NAME="2200"></A></para>
</sect2>
<sect2>
<title>TenDRA installation on Dec/Alpha</title>
<para><A NAME="5003">The TenDRA snapshot from November 1995 was the first
snapshot with support for the Dec/Alpha platform, but was not upward compatible
with the one installed on the Intel platform (TenDRA 4.0 versus TenDRA 3.1). So,
the TenDRA snapshot from November 1995 was installed on both the Intel/i386 and
the Dec/Alpha platforms.</A></para>
<para><A NAME="2174">DRA provide support for the DigitalUnix/Alpha platform, not
for the Linux/Alpha one. However, we benefited from the compatibility between
Digital Unix and Linux to solve this problem.We made three different
installations of the TenDRA technology for Alpha, among which the 2nd was fully
operational for Linux/Alpha:</A></para>
<para><A NAME="2151"></A></para>
<itemizedlist>
<listitem>First installation on native Digital Unix/alpha.
<A NAME="2152"></A></listitem>
<listitem>Second installation, still on Digital Unix, but for cross-development for
Linux/Alpha (termed <I>lin_alpha_cross</I>).
<A NAME="2153"></A></listitem>
<listitem>Third installation on Linux/Alpha (termed <I>lin_alpha</I>).
</listitem></itemizedlist>
<para><A NAME="2154">The first installation was straightforward and worked very
well. The main purpose was to ensure that the TenDRA technology worked correctly
on Dec/Alpha.</A></para>
<para><A NAME="2279">For the second installation, we created a new target
platform termed `lin_alpha_cross'. Most of the `lin_alpha_cross' files are
shared with Digital Unix, using symbolic links to directories or files, since
only a few files differ between the two targets. The main purpose of these
changes was to use Linux/Alpha system header files when compiling, instead of
the Digital Unix system header files, and Linux/Alpha libraries and startup
files when link-editing. For example, we changed three files inside the <I><target_platform>/private/env</I>
directory named <I>default</I>, <I>system</I> and <I>tcc_diag</I>. For the
same reason, we created a specific `lin_alpha_cross' subdirectory in lib/system
to hold some replacement system header files when cross-compiling with the <I>-Ysystem</I>
option (i.e. in DRA-NAT mode). The target dependent directories and files used
when (cross-)building APIs for Linux/Alpha, e.g. located under the <I>src/apis/libs</I>
directory, were also made specific.</A></para>
<para><A NAME="2291">Using this installation, we could cross-compile and
cross-link on Digital Unix for Linux, without any problem. The binary
compatibility between Linux/Alpha and Digital Unix was thus a key factor of
success.</A></para>
<para><A NAME="2156">The third TenDRA installation for Linux/Alpha was readily
derived from the previous one. We benefited again from the binary compatibility
with Digital Unix: we ran the TenDRA compilation chain, built for Digital Unix,
on top of Linux/Alpha, without the necessity to port it or recompile it.
However, to do so, we had to copy and install the shared library tools of
Digital Unix on Linux/Alpha because TenDRA uses shared libraries, for which
there is currently no support in Linux/Alpha. The Digital Unix shared libraries
mechanism works fine under Linux! This trick could have be avoided if we had
re-linked the TenDRA tools under Digital Unix using its statically-linked
libraries.</A></para>
<para><A NAME="2157">In order to use the Linux/Alpha native assembler and
link-editor instead of those from Digital Unix, we wrote a front-end shell
script to the TenDRA installer (trans). This shell script calls the actual trans
tool with an option to output a source assembly file instead of the `binary
assembly' files used by the Digital Unix as1 tool. Similarly, we wrote a
front-end shell script which emulates the call made by tcc to as1 by a call to
the Linux as tool. We give below the changes to the settings in <lin_alpha>/private/env/default
for the third installation:</A></para>
<para><A NAME="2158"></A></para>
<literallayout>+TRANS ``/..../linux/1.3.45/alpha/private/bin/trans.sh''
+AS1 ``/..../linux/1.3.45/alpha/private/bin/as1.sh''
+AS ``/usr/bin/as -nocpp'' # seems unused
+LD ``/usr/bin/ld -G8 -O1''
</literallayout>
<para><A NAME="2162">However, despite these modifications, the port of the TenDRA
installer to Linux/Alpha could not be completed. In some cases, the TenDRA
installer appeared to generate assembly instructions that are not recognized by
the Linux/Alpha assembler. For example, the following lines could not be
assembled properly by Linux/Alpha:</A></para>
<para><A NAME="2163"></A></para>
<literallayout>.extern __ctype_ 8
Error: Rest of line ignored. First ignored character is `8'
stq $fp, 8($sp)
Warning: Illegal operands
bis $17,$17,$fp
Warning: Illegal operands
.frame $fp, 360, $26, 0
Error: bad absolute expression; zero assumed
</literallayout>
<para><A NAME="2164">We now understand it is not surprising that we were unable
to use the TenDRA installer for Digital Unix/Alpha on Linux/Alpha. ANDF
installer output needs to be tailored for different target operating systems
according to the assembler and/or link editor interfaces supported by the target
operating system. In our case we attempted to use the assembler interface, and
the errors and warnings above are examples where this interface differs between
Digital Unix/Alpha and Linux/Alpha. Debugger support and even some details of
the procedure calling conventions may also need to be taken into account when
tailoring an ANDF installer to a different operating system.</A></para>
<para>Since we had already set up a cross-development TenDRA
environment for Linux/Alpha, hosted by Digital Unix, we continued to use it and
discontinued use of the `lin_alpha' installation. We actually installed TenDRA
on an NFS file server (used by the Linux/ i386, the Digital Unix and the
Linux/Alpha platforms).</para>
</sect2>
</sect1>
<sect1>
<title>5.3 Build environment with TenDRA</title>
<sect2>
<title>Definition of the set of Linux commands</title>
<para><A NAME="2647">The definition of the set of commands has not been done once
for all. It has been done on the Intel/i386 platform, during the first part of
the project, in two steps, level1 and level2, as described in </A><A
HREF="linux_re.htm#90">Description of Project phases</A>. Each step has been
performed as an incremental process. Each time new commands were selected, a
whole cycle of API definition (<A HREF="linux_re.htm#2743">§ 5.4</A>),
command ANDFization (<A HREF="linux_re.htm#2745">§ 5.5</A>), API
installation (<A HREF="linux_re.htm#6139">§ 5.6</A>) and command
installation and validation (<A HREF="linux_re.htm#1788">§ 5.7</A>) was
performed.</para>
<para><A NAME="2632">During the first phase of the project, a number of commands
have been compiled in DRA-DRA mode on the Unixware platform (see </A><A
HREF="linux_re.htm#6567">[2]</A>). We started by locating these commands in the
packages from the Slackware Linux source distribution for Intel/ix86. Since
these commands were among the simplest ones to ANDFize on Unixware, they were
good candidates to start with. Provided that a full binary installation was made
on our Linux/i386 platform, a command could be located in a package by searching
for its name in the list of packages under the<I> /var/adm/packages</I>
directory. This directory contains one text file per installed package, which
records the names of the commands it contains (actually the relative
installation path from / is provided for each command). When the name of the
package which holds a command has been found, we just had to connect to the ftp
site, find the directory of the same name in the Slackware source distribution,
and download the files under this directory.</para>
<para><A NAME="2495">Among the 103 Unixware commands ANDfized during the previous
phase of this project, we found 59 commands with similar name in Linux,
scattered among 11 packages: <I>bc, bin, bsdgames, diff, find, grep, gzip,
sh_utils, txtutils </I>and <I>util</I>. Moreover, these packages also contained
some additional commands which appeared to be good candidates for easy
ANDFization. However, we excluded a few of them which were compiled but not
delivered in the binary package, or which seemed too dependent on the target
platform (e.g. the
<I>fdformat</I> command which formats floppy disks). We also selected four
additional packages, <I>tar, cpio, xlock</I> and <I>xgames</I>, in order to
complete the <I>level 1</I> set of commands.</A></para>
<para><A NAME="2572">The definition of the <I>level 2</I> set of commands was
more difficult than for <I>level 1</I>, because we had to reject a number of
commands, for various reasons discussed below.</A></para>
<para><A NAME="11235">First, we tried to include in the <I>level 2</I> set of
commands more commands from X11, as we successfully experimented the ANDFization
of a few of them for<I> level 1</I>. However it appeared that this was not so
easy: the sources for these commands had not been packaged by Slackware, but
were provided inside a huge collection of sources named <I>Xfree86</I> (from
the Xfree86 Project, Inc.), itself derived from the X Consortium X11R6 code. An
attempt to perform the first step of the build of Xfree86 on Linux/i386, which
consisted in producing Makefiles from Imakefiles, failed. With some rewriting,
we managed to produce a Makefile for a simple Xfree86 command, <I>xclock</I>,
and successfully compiled it. However, we did not spend much time on
understanding the installation procedures of Xfree86, and, since it would have
taken us too much time per command to rewrite every Makefile, we set aside
Xfree86.</A></para>
<para><A NAME="2574">Then we excluded one package, <I>groff</I>, because it was
mostly coded in C++. We found also that some native Linux header files, used in
several packages, offer BSD compatibility but in a way that could not be
straightforwardly adapted to TenDRA. This issue is discussed in </A><A
HREF="linux_re.htm#2743">§ 5.4</A>.</para>
<para><A NAME="2577">We have included in the <I>level 2 </I>set of commands some
commands which represent quite large amounts of source code: <I>m4, elvis</I>
(a <I>vi</I> clone), <I>joe</I> (another editor), <I>less</I>, <I>perl</I>
and <I>elm</I>. The ultimate step of this experiment would have been to build
``monsters'' such as <I>bash</I> and <I>emacs</I>.</A></para>
<para><A NAME="2578">Finally, we evaluated the number of commands distributed
with a Linux system, and we found about 700 executable binary files in the
/usr/bin, /bin, /usr/X11/bin, /usr/openwin/bin, /usr/games, /sbin and /usr/sbin
directories. We examined some of these commands in the Slackware packages, and
concluded that there could be candidates for ANDFization. But we were limited by
time constraints to include such commands. Also, we did not port to the 2nd
platform, nor validate, all the commands operational on the 1st platform. Here
again, time is the main reason why we did not complete the port. We estimate
that an additional 1.5 engineer-month would have been sufficient to complete the
task, apart from some commands which may have been difficult to port.</A></para>
<para><A NAME="3621">The <I>level1 </I>and <I>level2</I> set of commands
include 236 commands: these commands were installed and validated on Linux/i386
(cf. section 4.7 on this point).</A></para>
<para><A NAME="6370">In the list below, the commands in <B>bold</B> (149) have
been installed and validated on both Linux/i386 and Linux/Alpha, as opposed to
the commands listed in plain characters, which are available on the first
platform only.</A></para>
<para><A NAME="6384">The few (13) commands listed in <I>italic</I> were ported
to both platforms, but their validation failed, or was not completed, on the
second platform.</A></para>
<para><A NAME="6390">Finally, in the `[p out of m]' statements, p is the
best-case number of commands we ported, while m is the maximum number of them
with respect to a given Slackware Linux package.</A></para>
<para><A NAME="3592"></A></para>
<itemizedlist>
<listitem>aaa_base package [9 commands out of 9]: fromdos, funzip, mtools, todos,
unzip, unzipsfx, zip, zipnote, zipsplit.
<A NAME="4856"></A></listitem>
<listitem>ash package [1 command out of 1] : ash.
<A NAME="3593"></A></listitem>
<listitem>bc package [1 command out of of 1]: <B>bc</B>.
<A NAME="3594"></A></listitem>
<listitem>bin package [48 commands out of 56]: at, <B>bban, bpe, chgrp, chmod, chown</B>,
compress, <B>cp</B>, crond, crontab, <B>ctags, dd, df,</B> dircolors, <B>du</B>,
ed, <B>elvis, elvprsv, elvrec</B>, file, fiz, <B>fmt</B>, <I>gawk</I>, <B>ginstall</B>,
indent, <B>ln, ls, man, mkdir, mkfifo, mknod, mv, patch, ref, rm, rmdir</B>,
sed, <B>shar, sysvbanner, time, touch, tput</B>, unarj, <B>unshar, uudecode,
uuencode, which</B>, zoo.
<A NAME="3595"></A></listitem>
<listitem>bsdgames package [13 command out of 36]: bcd, caesar, factor, fish, monop,
morse, number, paranoia, ppt, primes, rain, worm, worms.
<A NAME="3596"></A></listitem>
<listitem>byacc package [1 command out of 1]: <B>byacc</B>.
<A NAME="3597"></A></listitem>
<listitem>cpio package [2 commands out of of 2]: <I>cpio, mt-GNU</I>.
</listitem>
<listitem>diff package [4 commands out of 4]: <B>cmp, diff, diff3, sdiff</B>.
<A NAME="3599"></A></listitem>
<listitem>elm package [9 commands out of 9]: <I>answer</I>, <B>elm</B>, <I>elmalias</I>,
<B>fastmail, filter, frm,</B><I> newalias, newmail</I>, <B>readmsg</B>.
<A NAME="3600"></A></listitem>
<listitem>find package [6 commands out of of 6]: <B>bigram, code, find, frcode,
locate, xargs</B>.
<A NAME="3601"></A></listitem>
<listitem>flex package [1 command out of 1]: <B>flex</B>.
<A NAME="3602"></A></listitem>
<listitem>getty package [2 commands out of 2]: getty, uugetty.
<A NAME="3603"></A></listitem>
<listitem>grep package [1 command of 1]: <B>grep</B>.
<A NAME="3604"></A></listitem>
<listitem>gzip package [1 command out of of 1]: <B>gzip</B>.
<A NAME="3605"></A></listitem>
<listitem>ispell package [6 commands out of 6]: <B>buildash, icombine, ijoin,
ispell, sq, unsq</B>.
<A NAME="3606"></A></listitem>
<listitem>joe package [2 commands out of 2]: joe, termidx.
<A NAME="3607"></A></listitem>
<listitem>less package [2 commands out of 2]: <B>less, lesskey</B>.
<A NAME="3608"></A></listitem>
<listitem>m4 package [2 commands out of 2]:<B> ansi2knr, m4</B>.
<A NAME="3609"></A></listitem>
<listitem>perl package [4 commands out of 4]:<I> a2p, perl4.036, sperl4.036,
tperl4.036</I>.
<A NAME="3610"></A></listitem>
<listitem>ps package [11 commands out of of 12]: free, fuser, killall, ps, pstree,
psupdate, tload, uptime, vmstat, w.procps, w.bassman.
<A NAME="3611"></A></listitem>
<listitem>rcs package [8 commands out of 8]: <B>ci, co, ident, merge, rcs, rcsdiff,
rcsmerge, rlog</B>.
<A NAME="3612"></A></listitem>
<listitem>sh_utils package [24 commands out of 24]:<B> basename, date, dirname,
echo, env, expr, id, logname, nice, pathchk, printenv, printf, pwd, sleep, stty,
su, tee, test, tty, uname, users, who, whoami, yes</B>.
<A NAME="3613"></A></listitem>
<listitem>sudo package [2 commands out of 2]: sudo.bin, visudo.
<A NAME="3614"></A></listitem>
<listitem>tar package [3 commands out of of 3]: <B>tar</B>, <I>rmt</I>, testpad.
<A NAME="3615"></A></listitem>
<listitem>tcpip package [7 commands out of 31 (from the net-tools subset)]: arp,
ifconfig, plipconfig, rarp, route, netstat, slattach.
<A NAME="3616"></A></listitem>
<listitem>txtutils package [22 commands out of 22]: <B>cat, cksum, comm, csplit,
cut, expand, fold, head, join, nl, od, paste, pr, sort, split, sum, tac, tail,
tr, unexpand, uniq, wc</B>.
<A NAME="3617"></A></listitem>
<listitem>util package [35 commands out of 57]: agetty, <B>arch, banner, chfn,
chroot, chsh, col, colcrt, colrm, column, ddate, frag, hexdump, hostname, ipcrm,
ipcs, last, login, mesg, more, newgrp, passwd, rdev</B>, readprofile, <B>renice,
rev, setsid, sln</B>, strings, swapon, <B>ul, vipw, wall</B>, <I>zdump</I>,
<B>zic</B>.
<A NAME="3618"></A></listitem>
<listitem>xgames package [8 commands out of 13] : maze, xcolormap, spider, xtetris,
xlander, xminesweep, xroach, xvier.
<A NAME="3619"></A></listitem>
<listitem>xlock package [1 command out of 1]: xlock.
</listitem></itemizedlist>
<para><A NAME="4846"></A></para>
</sect2>
<sect2>
<title>Setting up the build environment</title>
<para><A NAME="4845">The environments for the NAT-NAT, DRA-NAT and DRA-DRA builds
have been setup using similar to those used during the Unixware port.</A></para>
<para><A NAME="2815"></A></para>
<itemizedlist>
<listitem>One single reference source tree, then a dedicated work tree per (build,
target platform). For the 1st target (Linux/i386), each work tree holds symbolic
links to the <I>source</I> tree, while binaries are built inside a <I>work</I>
tree as plain files. In addition, a procedure is used to replace a link to the
source tree by a link to a <I>patch</I> tree when a source file has to be
modified during the port to TenDRA. This is very similar to the environment we
had on Unixware. The major difference is that each package has its own set of
<I>source/work/patch</I> file trees. This is more modular but requires more
manipulations.
<A NAME="8189"></A></listitem>
<listitem>For the 2nd target (Linux/Alpha): we usually created only a work tree for
the `DRA-DRA' build. It initially contained `source' files only, which are
symbolic links to their equivalent in the `DRA-DRA'/i386 work tree. By `source
files' we mean here the Makefiles and the <B>ANDF</B> - <B>.j</B> - files
having been generated from the original .c files by the TenDRA producer, during
the DRA-DRA build for Linux/i386.
<A NAME="8208"></A></listitem>
<listitem>A shell script used as a pseudo cc (e.g. pseudo gcc) during the DRA-NAT and
DRA-DRA builds. This avoids the necessity to modify most of the original
makefiles when building the commands. The pseudo <I>cc</I> used during the
build for the 2nd platform substitutes the (usually unique) <I>input_file</I><B>.c</B>
by <I>input_file</I><B>.j</B>.
</listitem></itemizedlist>
<para><A NAME="2809">One specific feature of the sources and build procedures of
the Linux commands is that they have often been designed to support a variety of
target platforms and UNIX variants at source level. Thus, when building a
command for the first time, there is usually a preliminary self-configuration
step which examines the system header files, and produces a local header file
(or a customized Makefile) which summarizes the target system peculiarities by
means of #define (or -D) statements. We ran such self-configuration scripts
before creating the NAT-NAT, DRA-NAT and DRA-DRA work trees: this assumes that
our second platform for porting (Linux/Alpha) is to provide similar APIs to the
1st one (Linux/i386). Eventually, we had to revise the settings chose by the
self-configuration.</A></para>
<para><A NAME="2801"></A></para>
</sect2>
<sect2>
<title>NAT-NAT/i386 and DRA-NAT/i386 build problems</title>
<para><A NAME="2827">These two builds of the commands were only performed on the
Linux/i386 platform, as a sanity check and cleanup of the source code.</A></para>
<para><A NAME="2917">We faced only one problem during the NAT-NAT/i386 build of a
few commands.</A></para>
<para><A NAME="2845"></A></para>
<itemizedlist>
<listitem>Some header files (e.g. <I>linux/autoconf.h</I>) were not found when
attempting to compile some administrative commands. To gain access to such
headers, the preliminary step of a kernel rebuild can be done; alternatively,
one could manually establish the proper symbolic links for <I>/usr/include/linux
</I>and
<I>/usr/include/asm</I>: they should point to their equivalent inside the<I>/usr/src/linux/include
</I>directory.
</listitem></itemizedlist>
<para><A NAME="2849">We faced a limited number of problems during the
DRA-NAT/i386 builds of the commands. We list these problems below:</A></para>
<para><A NAME="2829"></A></para>
<itemizedlist>
<listitem>The link-edit of some commands failed because one symbol was undefined:
<I>_alloca</I>. In the native compiler (gcc), <I>_alloca</I> is implemented as
a built-in function. In the TenDRA compiler, this can also be the case, provided
that the header file <I>alloca.h</I> is explicitly included. So we modified the
relevant source files to include this header file.
<A NAME="2842"></A></listitem>
<listitem>The source code for some commands appeared to use, through the inclusion of
a system header file or under <I>#ifdef i386</I> conditional instruction, some
assembly code. The related commands were thus excluded from our set of commands,
except for a few of them for which we found a C variant to the assembly code.
<A NAME="2592"></A></listitem>
<listitem>Re-declaration of an array, for which the dimension was computed using
<I>sizeof</I>. The following code sums-up the problem:
<A NAME="2593"></A>
<literallayout> extern int lnum[sizeof(short)];
int lnum[sizeof(short)]; /* bis */
</literallayout>
</listitem></itemizedlist>
<para><A NAME="2595">We sent a Change Request, <I>array_sizeof(262)</I>,
concerning this problem, which applied to the apr-95 and nov-95 TenDRA releases.
It has now been fixed.</A></para>
<para><A NAME="2597"></A></para>
<itemizedlist>
<listitem>Name conflict between a function and its arguments. The following code
sums-up the problem:
<literallayout> char *fields(fields)
char *fields;
{ return fields; }
</literallayout>
</listitem></itemizedlist>
<para><A NAME="2901">We sent a Change Request, <I>func_var(262)</I>, concerning
this problem, which applies to the apr-95 and nov-95 TenDRA releases. It has now
been fixed.</A></para>
<para><A NAME="2603"></A></para>
<itemizedlist>
<listitem>Use of custom options of the native compiler (gcc), e.g.:
<literallayout> -fpcc_struct_return</literallayout> This option was used in the Makefile for the
getty package. The gcc man page says that this option provides intercallability
with modules (e.g. library modules) compiled with a pcc compiler. We concluded
that this was not relevant when compiling for a Linux target platform, since gcc
is used to compile the libraries, and we ignored it. Similarly, we ignored, i.e.
filtered out in our pseudo-gcc for DRA-NAT/DRA-DRA builds, many other gcc
options such as:
<literallayout> -fomit-frame-pointer, -pipe, -g</literallayout>while we adapted to TenDRA style
some others, such as:
<literallayout> -static /* gcc */ --> -Wl,-static /* tcc */
</literallayout> </listitem></itemizedlist>
<para><A NAME="2743"></A></para>
</sect2>
</sect1>
<sect1>
<title>5.4 Definition of the API for the commands</title>
<para><A NAME="5031">We started the experiment with an xpg3 API, and decided to
put all other symbols we needed in an extension API. However, after a few
compilations of Linux commands, it became clear that most of the symbols we were
adding to the extension API were in fact part of some other standard APIs, such
as svid3.</A></para>
<para><A NAME="3057">So, we redefined our base API to be a merge between the
xpg3, svid3, gcc and bsd_extn APIs delivered with TenDRA, limiting the extension
API to symbols specific to the Linux commands interface. In fact, some of the
symbols in the extension API are defined in the standard cose API, but since
this API is very partially supported by Linux, and sometimes conflicts with
definitions provided in other APIs, it was not worth including it in the base
API.</A></para>
<para><A NAME="3085">For the<I> level 2</I> set of commands, we downloaded some
packages using a BSD-like interface, and we tried to include the symbol
definitions for these commands in our extension API. However, this appeared to
be very difficult, since we found that the Linux implementation of some BSD
interfaces redefines symbols from the POSIX API, in an incompatible way. This is
reflected in the Linux header files by conditional definitions, selected with
the _BSD_SOURCE macro for example, or by replacement header files, such as
bsd/signal.h instead of signal.h. The incompatible definitions we found were for
the <I>jmp_buf</I> type, the <I>setjmp()</I>, <I>getpgrp()</I>, <I>wait()</I>,
<I>waitpid()</I>, <I>wait3()</I> and
<I>wait4()</I> functions, and finally the<I> signal()</I> function redefined
as <I>bsd_signal()</I>.</A></para>
<para><A NAME="3154">This problem could have been resolved by removing from our
base API the conflicting symbols, and creating a <I>conflict_posix</I> and a
<I>conflict_bsd</I> extension APIs with these symbols. The compilation of the
commands based on a BSD-like API would have used the <I>conflict_bsd</I> API,
in addition to the base and `regular extension' APIs, and would have been
link-edited with the
<I>libbsd</I> library provided by Linux. Since this would have taken a lot of
time, we preferred not to modify our API and we set aside these commands, unless
we found a simple work-around: selecting at build-time, or recoding to, a POSIX
adherence for them (refer to next section).</A></para>
<para><A NAME="3069">Finally, in order to compile some X11 commands, a separate
API including the <I>x5_lib</I>, <I>x5_t</I>, <I>x5_mu</I>,<I> x5_aw</I> and<I>x5_mit</I>
standard APIs, has been created. Since Linux is based on X11R6, an extension API
has also been created, which includes the few symbols we had to define for the
X11 commands we built.</A></para>
<para><A NAME="5289">We found one inconsistency between the Linux header files
and the standard API provided with TenDRA for the <sys/socket.h> header
file, defined in the bsd_ext API: we had to change almost every use of <I>caddr_t</I>
to <I>struct sockaddr *</I>. We also found a few inconsistencies between the
Linux/i386 and Linux/Alpha header files, which have been resolved by some
corrections to the Linux native header files, in the API definition, and in the
source code for one command (<I>more</I>).</A></para>
<para><A NAME="2745"></A></para>
</sect1>
<sect1>
<title>5.5 ANDFization of the commands</title>
<para><A NAME="5033">We encountered different kinds of problems when compiling
with TenDRA the set of commands on the Linux/i386 platform. Among these
problems, only one was related to a bug in the TenDRA technology, the others
were either related to ANDF constraints, or to more general portability issues.</A></para>
<para><A NAME="6085"></A></para>
<sect2>
<title>Dealing with ANDF constraints</title>
<para>We list below problems we encountered while ANDFizing Linux
commands, which are related to the use of the TenDRA technology as a
replacement
to a classic compiler. We start with the only bug found in TenDRA during
this
process, then we roughly follow the order in which the various issues were
encountered.</para>
<itemizedlist>
<listitem>
<para>Redefinition of an API token as a macro</para>
<para>In the code below, <I>alarm</I> is defined as a macro, but it is
also a token in our API:</para>
<literallayout>
#include <unistd.h> /* for alarm() */
xtern int debug() ;
#define D_RUN 1
#define alarm(d) alarm(d); debug(D_RUN, ``alarm set: %s:%u'',\
__FILE__, __LINE__)
long xx() { return alarm((long)5); }
</literallayout>
<para>This code is indeed illegal, but tdfc entered an infinite loop.
The problem was reported to DRA as <I>loop_tdfc_alarm(276)</I>, and
has now been fixed.</para>
</listitem>
<listitem>
<para>Added missing startup macros<para></para>When the TenDRA
compiler (tcc) was used, a number of startup flags, defined with the
native compiler (gcc), were missing. The <I>linux</I>,
<I>__linux__</I>, <I>unix</I> and _<I>_X11_P_HEADERS</I> flags, plus a
number of flags defined in the native <I>features.h</I> header file,
such as <I>_POSIX_SOURCE</I>, were added in a startup file for tcc.
</para>
</listitem>
<listitem><para>Added missing function prototypes and fixed type
mismatches</para><para>We used a tcc option to warn about missing function
prototypes, and we fixed them by either including the appropriate header
files or adding their prototype for locally defined functions. We added
casting on some calls to library functions. Then, every remaining
undeclared symbol was added to the extension API. <emphasis>Note than more
than half of the changes we made in the source code for Linux commands
consisted in adding such prototypes.</emphasis>
<A NAME="3799"></A></para></listitem>
<listitem>
<para>Resolved one conflict with an API symbol</para>
<para>The function <I>mkdir()</I>, local to the file <I>mtools</I>, has been
renamed to avoid a conflict with the API symbol defined in
<sys/stat.h>.<A NAME="5412"></A></para></listitem>
<listitem>Illegal use of target-dependent condition<BR/>In the code below,
INT_MAX is a target dependent token, which cannot be used to conditionally
define a preprocessor macro:<A NAME="5413"></A>
<literallayout> #if (INT_MAX <= 65535)
#define longdiff(a, b) /* (definition 1 for the macro) */
#else
#define longdiff(a, b) /* (definition 2 for the macro) */
#endif
</literallayout>
<I>We fixed it by replacing the macro definition by a static functions:</I>
<literallayout> static int longdiff(time_t a, time_t b) {
#if (INT_MAX <= 65535)
/* ... (definition 1 for the function) */
#else
/* ... (definition 2 for the function) */
#endif
}
</literallayout>
<para>This constraint arises from the way ANDF is used to achieve
portability between targets which may have different values for
INT_MAX. The constraint is that a target-dependent #if is permitted
only where a statement is permissible, and both alternatives must be
legal statement lists.</para>
<para>This constraint unfortunately prevents target-dependent macro
definitions in the style shown above. DRA is currently considering
whether the constraint may be eased in a subsequent version of TenDRA
to permit certain well-formed cases such as this.</para>
</listitem>
<listitem>
<para>POSIX.1 or SVID interfaces versus BSD interfaces</para>
<para>Three functions of the <I>bin</I> package, <command>time</command>
and <command>crontab</command>, and <command>ash</command> (a simple
shell), were configured to use some BSD interfaces which had not been
included into our API, as discussed in
<link linkend="linux_re.htm#2743">§ 5.4</link>.</para>
<para>For the <command>time</command> command, we found that the support
for POSIX interfaces was provided in the source code, so we used it.
</para>
<para>Similarly, prior to building <command>ash</command>, we modified
the related configuration file and Makefile, in order to elect
svid3-like interfaces instead of the default bsd ones.</para>
<para>For the <command>crontab</command> command, we fixed the problem by
removing in the source code some (simple) calls to the BSD
<function>wait4()</function> function, and by using the XPG3
<function>waitpid()</function> function instead.</para></listitem>
<listitem>For a few commands which use the <I>curses</I> interface, e.g.
<I>bpe</I>, we chose the svid3 variant instead of the BSD one (they are
both supported by Linux). Makefiles for building these commands with
TenDRA have been changed to use the <I>libncurses</I> library instead of
the <I>libcurses</I> library at link-edit time. Note that the sources for
the <I>elvis</I> editor (from the <I>bin</I> package) embed a small custom
version of the <I>curses</I> interface.</listitem>
<listitem>For some commands, the initial self-configuration step performed
prior to entering the actual build defines the path for another command,
because the latter is called by the first one by means of
<function>exec()</function> or <function>system()</function>. While Linux
provides the <paths.h> header file for this purpose, we found some
files which do not include this header file, and others which need to call
a command for which there is no path definition in the regular header. An
example of such a situation is <command>elm</command>, which calls an
editor (e.g. <command>vi</command>). When we detected such situations, we
either modified the source code to include and use <paths.h>, or we
added a definition inside the alternate <paths.h> specified in our
extension API.</listitem>
<listitem>Pointer/Integer conversion<BR/>The TenDRA compiler can be
configured to issue a warning on every pointer/integer conversion. This is
done with the pragma instruction:<BR/><I>#pragma TenDRA conversion
analysis (int-pointer) warning</I><BR/>However, due to the very large
number of occurrences of these warnings, we had to cancel this mode, and
decided to postpone their analysis until after the validation
step.<BR/>For example, we encountered uses of `-1' (minus one) to give a
special meaning to a pointer value, while only `0' (NULL) is accepted for
this purpose. (Note: 64-bit issues are discussed later in this section.)
<A NAME="5432"></A></listitem>
<listitem>Underspecified type in svid3 API<BR/>We found one command which
makes the assumption that the daddr_t type, defined in the svid3 API, is an
arithmetic type. The source code casts a daddr_t value into an int, while
daddr_t is defined in the API as:
<literallayout>
+TYPE daddr_t;
</literallayout>
We fixed this problem by stating that daddr_t is indeed an arithmetic type,
which is correct for both Linux/i386 and Linux/Alpha. We initially modified
the reference svid3 API, but later we did it more cleanly, moving the
daddr_t definition to our extension API prior to changing it to:
<literallayout>
+TYPE (int) daddr_t;
</literallayout>
Also, prior to fixing this in the API, we found that casting an integer
value to a daddr_t type was not rejected by the TenDRA compiler, while it is
obviously illegal. A bug report has been sent to DRA, and this has now been
fixed.
</listitem>
<listitem>Recoding of a source file dealing with target platform byte ordering
issues.<BR/>A source file used a local BYTEORDER macro, set-up during the
initial self-configuration step of the build of the command, to support
different byte ordering. However, Linux already provides for this purpose a
__BYTE_ODER macro, defined in the <bytesex.h> header file. So, we added
the __BYTE_ORDER symbol to our API, and replaced all the occurrences of the
BYTEORDER macro by references to the __BYTE_ODER API macro. We also had to
rewrite some code, because with TenDRA some instructions are illegal after a
target dependent condition.</listitem>
<listitem>The termio and termios interfaces are both provided with Linux,
and share the same set of macros to define indexes in the<I> c_cc </I>array
from either the termio or termios structures. On Linux/i386, these indexes
are the same for the two structures, while on Linux/Alpha they differ.
TenDRA provides a way to support different variants of a same object, using
version numbers, and this should have solved our problem. However, since we
never used this feature before, we did not spend time to see how we could
use it in our API. Instead, we made a temporary fix, consisting in renaming
the constants in the termio interface. For two of the commands we ported,
<I>more</I> and <I>ispell</I>, which use the termio interface, we changed
their sources to reference the new macros.
</listitem></itemizedlist>
</sect2>
<sect2>
<title>Undocumented dependencies to the OS / the underlying hardware</title>
<para>Some commands are platform dependent, and are not
easily (sometimes: not at all) portable from one platform to another.
However, the Linux/i386 and Linux/Alpha OS's are very similar;
furthermore, some hardware architectures built around the DEC Alpha chip
are not much different from the Intel-based PC: for example, our
Linux/Alpha platform includes ISA adapters for graphics and Ethernet. In
such a favorable situation, the Linux/i386 <I>ifconfig</I> command (from
the<I> tcpip/net-tools</I> subset), which displays hardware information on
the network interfaces such as their (ISA) ``Base address'', could
probably have been easily ported to Linux/Alpha (/AXP pci). Also, the perl
command, which includes optional support for undocumented system calls,
may or may not be portable between two Linux platforms, depending on the
system calls they implement.</para>
<para>On the other hand, changing the format for binary
files, e.g. switching from a classic Linux <I>a.out</I> format to the
Digital Unix `Extended COFF', may require changes in some common commands
such as <I>strings</I> or <I>file</I>. When using a classic compiler, some
(or even all in a favorable case) of these changes may be hidden inside the
system header files, e.g. <a.out.h>, but the TenDRA compilation chain,
when used in DRA-DRA mode, is often more rigid.</para>
</sect2>
<sect2>
<title>Upgrade to TenDRA 4.0</title>
<para>When we had ANDFized the whole set of commands, we
upgraded to TenDRA 4.0 in order to work on the Linux/Alpha platform (see
<A HREF="linux_re.htm#2200">TenDRA installation on Dec/Alpha</A>).
However, we had to ANDFize again the set of commands with the new TenDRA
version, since it was not upward compatible with the previous one. In
fact, we only re-ANDFized the commands we tried to install on the
Dec/Alpha platform, at the time we needed them. We did not encounter any
problem when doing this.</para>
</sect2>
<sect2>
<title>Holes in source code portability (64-bit vs 32-bit issues)</title>
<para>During the installation and validation of these
commands on the second platform, we found a number of bugs related to
portability problems, which are out of the scope of TenDRA. All these bugs
were due to code assuming 32-bit platforms, which break on 64-bit
platforms. Some of the bugs we found were already fixed in early
Linux/Alpha releases, such as the Blade release, others were still there.
We fixed the source of the commands, re-andfized them, and installed and
validated these commands again on the two platforms. We give below the
portability issues we encountered:</para>
<itemizedlist>
<listitem>Wrong <I>int <-> pointer</I> conversion<BR/>On Linux/Alpha
(and Digital Unix/Alpha), a pointer type is 8-bytes wide, so it cannot
fit in an <I>int</I> type, which is only 4-bytes wide. Fortunately, the
<I>long</I> type on Linux/Alpha is, as usual, as large as a pointer, and
thus can be used as a replacement for an <I>int</I>, each time an
explicit pointer<->integer is used. This is a common type of the
portability fixes we had to make in the source code for Linux commands,
subsequent to encountering a `Linux/Alpha-only' problem at validation
time.</listitem>
<listitem>Incorrect assumptions on sizes of <I>int</I>, <I>long</I> and
<I>size_t</I><BR/>Although in many cases int and long types are equivalent,
we give below three examples of code we found where it makes a difference:
<literallayout>
/* #1 */ { int i; printf(``i value is: %ld\n'', i); }
/* #2 */ extern char *malloc(int);
/* #3 */ { long l ; printf(``%08lx'', l); }
</literallayout>
All three work perfectly on Linux/i386, while they cause, or could
cause, damage under Linux/Alpha.<BR/>In the first two lines the
function, printf or malloc, will read respectively a long on the stack,
which are 8 bytes wide, while only 4 bytes for an int would have been
pushed. Note that the correct prototype for malloc is char
*malloc(size_t), and that size_t is equivalent to long on Linux/Alpha.We
fixed the error on such printf statements with a cast to long for an
argument, and the error on malloc by replacing its local-and-wrong
declaration by the inclusion of the <stdlib.h> header
file.<BR/><BR/>In the third case, the instruction was used to print a
fixed number of digits. However, a long, 8-bytes on an Alpha platform,
may hold a value that prints up to 16 digits, thus putting unexpected
digits in the output. In the code where we found the problem, the fix
was to truncate the value to a 4-byte value. </listitem>
</itemizedlist>
<para><A NAME="6139"></A></para>
</sect2>
</sect1>
<sect1>
<title>5.6 Installation of the API for the commands</title>
<sect2>
<title>Installation of the API on Linux/i386</title>
<para>The API is made of two parts, a base API, which is a
merge between xpg3, svid3, gcc and bsd_extn APIs, and an extension API,
which completes the interface required for the commands (see <A
HREF="linux_re.htm#2743">Definition of the API for the
commands</A>).</para>
<para><A NAME="4431">The base API was installed on the Linux/i386 platform,
without any problem. However, we left some tokens undefined when some parts of
the API were not part of the actual Linux API. Then, the extension API was
installed, as we extended it to cover more and more commands. These
installations required a few patches to the system header files, some of which
were already provided with the TenDRA snapshot. These installations went very
well, with only a few problems, listed in a following paragraph. Then, when we
moved to a new TenDRA snapshot to the Linux/Alpha platform, we re-installed the
API, without any problem.</A></para>
<para><A NAME="5724">We made a small number of modifications to the API during
the port of the commands on the Linux/Alpha platform. However, each modification
we made required re-installation of some parts of the API.</A></para>
<para><A NAME="4094"></A></para>
</sect2>
<sect2>
<title>Installation of the API on Linux/Alpha</title>
<para><A NAME="4163">As for the Linux/i386 platform, we had to apply some patches
to the system header files in order to install the API. A number of these
patches were actually identical to the patches we made on the Linux/i386 header
files. So, instead of copying and editing, by hand, these files again, we chose
to implement these patches by means of <I>sed</I> scripts, which could be
applied to both Linux/i386 and Linux/Alpha header files. Most of these scripts
are now common to the two platforms, although a few of them are specific to one.
These
<I>sed</I> scripts not only facilitate corrections to the system header files,
but would also be useful if we need to upgrade from one Linux version to
another.</A></para>
<para><A NAME="4448">The Linux/Alpha system header files do not differ much from
the Linux/i386 ones. However, since the Linux/Alpha port was derived from a more
recent Linux/i386 version (Linux 1.3) than the one we used (Linux 1.1), we could
not clearly distinguish between the changes which come from standard Linux
evolutions and those which have been introduced during the port of Linux to
Digital Alpha. One important modification was that some definitions found on
Linux/i386 in some <linux/*> or <sys/*> header files have now been
moved into <asm/*> header files. We had to take such changes into account
when adapting the Linux/i386 modifications to Linux/Alpha.</A></para>
<para><A NAME="4449">Finally, we did not port to Linux/Alpha the extension API to
the TenDRA x5/* APIs. This would have required installation of X11 on our
Linux/Alpha box, which consists of 28 additional floppy images! This part of the
API was only required for 9 commands from our set of commands, which we did not
install on Linux/Alpha.</A></para>
<para><A NAME="4199"></A></para>
</sect2>
<sect2>
<title>API installation problems</title>
<para><A NAME="4211">We list below the problems we found when building the API on
the Linux/i386 or the Linux/Alpha platforms, and the solution we adopted.</A></para>
<para><A NAME="11427"></A></para>
<itemizedlist>
<listitem>A macro, <I>makedev</I>, added to our extension API, was defined in the
Linux/i386<I>sys/sysmacros.h </I>header file. This file contains the lines:
<literallayout> #define major(dev) ...
#define minor(dev) ...
#define makedev(major,minor) ...
</literallayout> The identifiers <I>major</I> and <I>minor</I> used to name the formal
parameters of the <I>makedev</I> macro are also the name of two macros defined
in this header file, which we included in our API. The clash on the names is
reported as an error by tdfc when building the API. We did not check whether
this was a TenDRA bug or not, but we bypassed the problem by using an alternate
version of <I>sys/sysmacros.h</I>, in which the formal parameters of the <I>makedev</I>
macro have been renamed.
<A NAME="4269"></A></listitem>
<listitem>The <bytesex.h> header file contained the following lines:
<literallayout>
#undef __BYTE_ORDER
#define __BYTE_ORDER 1234
</literallayout> The<I> #undef</I> line prevents tcc from finding the definition of the
__BYTE_ORDER constant. This is a constraint that applies only when building API
token libraries. It is a necessary consequence of using C macro definitions to
obtain ANDF token definitions. We bypassed this behaviour by commenting out the
#undef line in a replacement header file.
<A NAME="4274"></A></listitem>
<listitem>The Linux/Alpha header files do not follow the standard APIs in some cases.
For example, the Linux/Alpha <sys/stat.h> header file defines the field
<I>st_dev</I> from the <I>stat</I> structure as `unsigned int' instead of
`dev_t' as defined in XPG/3. Since `dev_t' is equivalent to `unsigned int' on
Linux/Alpha, we were able to modify the system header file to use the correct
type.
<A NAME="4318"></A></listitem>
<listitem>The Linux/Alpha header files sometimes use an `int' type, in places where a
`long' type is used on Linux/i386. In such cases, we decided to patch the
Linux/i386 header files to use an `int' type, since `long' and `int' are
equivalent on a 32-bit platform.
<A NAME="4337"></A></listitem>
<listitem>The reverse situation, where Linux/i386 uses an `int' and Linux/Alpha uses
a `long', has also been found. In this case, we preferred to modify the API to
accept both types, using the tspec `+TYPE (int) ...' notation.
<A NAME="4406"></A></listitem>
<listitem>On the Linux/i386 platform, we extended the API with some symbols from the <termio.h>
header file. The <termio.h> system header file has changed between
Linux/i386 and Linux/Alpha, and some definition were incompatible with the
extension API. We eventually found a solution, which involved a fix to the Linux
header files.
<A NAME="4361"></A></listitem>
<listitem>The Linux system supports two variants of the <I>curses</I> library, one
defined by the <curses.h> header file for a BSD API, and the other defined
by the <ncurses.h> header file for a svid3 API. We used the latter to
build the API, since we do not support the BSD API.
<A NAME="5348"></A></listitem>
<listitem>The TenDRA svid3 API defines the constant RLIM_INFINITY, from the
sys/resource.h header file, as follow:
<literallayout> +CONST int RLIM_INFINITY;
</literallayout> However, this constant is used to assign variables of type <I>rlimit_t</I>,
which is, on Dec/Alpha, defined as a long, thus 8-bytes wide. So the problem
was: while this constant was defined with the value 0x7fffffffffffffffL, it was
actually truncated to fit within a (32-bit) int. We fixed this bug by replacing
the definition of RLIM_INFINITY by:
<literallayout> +CONST rlimit_t RLIM_INFINITY;</literallayout>
</listitem></itemizedlist>
<para><A NAME="1788"></A></para>
</sect2>
</sect1>
<sect1>
<title>5.7 Installation and validation of the commands</title>
<para>On the Linux/i386 platform, we used the TenDRA compiler to
produce the ANDF files, and then translate them into binary executable files, in
the same invocation of the compiler. A tcc option was used to preserve the
intermediate ANDF files.</para>
<para><A NAME="4569">On the Linux/Alpha platform, we used the ANDF files produced
on the Linux/i386 platform, and translated them into binary executable files.
For this platform, we ran the TenDRA compiler on a Dec/Alpha platform as a
cross-compiler for the Linux/Alpha platform (see </A><A HREF="linux_re.htm#2200">TenDRA
installation on Dec/Alpha</A>).</para>
<para><A NAME="4627">In order to validate the commands we built, we used several
different methods, depending on the commands.</A></para>
<para><A NAME="6233">We found that a very limited number of commands were
packaged with some rather extensive self-validation tests, that we used to
validate such commands.</A></para>
<para><A NAME="6234">We tested some other commands interactively (e.g. <I>elvis</I>,
<I>bpe</I>, <I>ispell</I>, <I>elm</I>, ...). However, for most commands we
had to write small tests. Even a basic test requires several shell script lines:
it took us several weeks to write tests for >200 commands then to run them.</A></para>
<para><A NAME="6235">Finally, a small number of commands, actually 10, was not
tested: <I>getty/uugetty</I>, <I>sudo.bin/visudo</I>, <I>readprofile</I>,
<I>swapon</I>, <I>mt-GNU</I>, <I>rmt</I>,
<I>plipconfig</I> and <I>slattach</I>. As none of these commands were actually
installed on the 2nd platform, there is no real penalty.</A></para>
<para><A NAME="4669">On the Linux/i386 platform, 236 commands were installed,
then validated. Conversely, on Linux/Alpha, we installed and validated only a
subset of these commands: about 150, as previously mentioned. While we found
only a few problems during the validation on Linux/i386, we faced a number of
validation failures on Linux/Alpha, thus requiring much more investigation. Some
of these problems were due to bugs in the TenDRA technology, and have all been
fixed by now. Most of the others have already been discussed in previous
sections.</A></para>
<para><A NAME="5957">We list below miscellaneous problems, encountered at
validation time on either the 1st or the 2nd platforms, which are not really
related to TenDRA, nor do they depend on portability issues in the original
source code.</A></para>
<para><A NAME="4579"></A></para>
<sect2>
<title>Miscellaneous problems encountered at validation</title>
<para><A NAME="4671"></A></para>
<itemizedlist>
<listitem>For two commands, <I>sln</I> and <I>stty</I>, we found different
behaviour due to an undetected missing symbol in our API. For example, the <I>sln</I>
command is written such that, depending on weather the S_ISLINK symbol is
defined or not, it generates a symbolic link or a hard link. The missing symbol
was defined in our API to fix the problem.
<A NAME="8290"></A></listitem>
<listitem>The <I>more</I> command on Linux/Alpha, of which the source code uses the
non-POSIX <I>termio</I> interface, waits for 4 input characters before
processing them. The reason is that in Linux/Alpha (as in Digital Unix) some of
the <I>termio.c_cc[]</I> control characters overlap with some others, depending
on the input mode being used (i.e. either the `canonical' mode or the
`half-cooked' mode). So such control characters, which are the ones at VMIN and
VTIME indexes, must always be updated when switching from one input mode to the
other: we changed the source code for the <I>more</I> command accordingly.
<A NAME="5516"></A></listitem>
<listitem>We found two bugs in the Linux/Alpha libc library, in the <I>getegid</I>
and <I>times</I> functions. The 1st one was already known and fixed in a later
release, while the 2nd one was not (we received a fix for it a few days later,
from a member of the Linux/Alpha project).
<A NAME="4522"></A></listitem>
<listitem>The <I>csplit</I> command, from the txtutils package, defines the <I>memchr()
</I>function, also defined in the libc library. The validation of this command
failed on Alpha, until we removed this local recoding for <I>memchr()</I> and
used the regular library entry point instead.
<A NAME="13121"></A></listitem>
<listitem>The environment on Linux/Alpha for using the <I>zic</I> command was not
correct, since that command is not available in the Linux/Alpha BLADE_0.3
distribution.
<A NAME="13122"></A></listitem>
<listitem>The environment on Linux/Alpha for running the <I>bpe</I> command was not
correct, since we had used the (available) <I>ncurses</I> library when
link-editing this command: at run-time the related <I>terminfo</I> `data base'
was lacking. We simply had to replicate on Linux/Alpha the Linux/i386 terminfo
files to cure this problem.
</listitem></itemizedlist>
<para><A NAME="13123"></A></para>
</sect2>
<sect2>
<title>Recent upgrades of our original source code for Linux commands</title>
<para><A NAME="6280">We initially found on the net a limited number of patches
for the commands in source form; such patches had been made during the port of
Linux to Digital Alpha. For example, the following patch for the source code for
the col command was found on <I>ftp.azstarnet.com</I>, but it is not a example
of interest since we had preventively fixed it, i.e. during the initial
ANDFization of the col command: </A></para>
<para>util-linux-2.5/text-utils/col.c:</para>
<literallayout> + #include <malloc.h></literallayout>
<para>Conversely, neither the BLADE_0.3 nor the BLADE_0.2
distributions include source code for commands; we discovered very
recently that such code was available (on <I>ftp.digital.com</I>)
for the very first, 32-bit Linux/Alpha distribution. Nevertheless,
for three commands we used source code from the RedHat 2.1/beta
and 2.1 Linux/Alpha distributions. This allowed us to fix
incorrect behavior of the <I>zic</I> command; we also experimented
without success (due to lack of time) some partial upgrades of the
source code for <I>zdump</I> and <I>cpio</I>.</para>
</sect2>
</sect1>
</chapter>
<chapter>
<title>6. Statistics</title>
<sect1>
<title>6.1 Statistics for Linux APIs</title>
<para>All the files required to create an API are located inside
a special sub-tree of the TenDRA [4.x] distribution named<I> src/apis/</I>.
First, the new API must be specified, in a dedicated language (the TenDRA <I>tspec</I>
tool is provided to translate such API specification into a target-independent
intermediate format). An API specification is split into files, each of which
usually corresponding to a system header file available on the target
platform(s); e.g.: <I>stdio.h, sys/types.h</I>, etc.</para>
<para>Secondly, there is often the need for slight changes within
some of the system header files of a given target, in order to build (to
install) the new API on it; such changes are either hard-coded inside a
replacement version for the relevant headers, or automatically performed using
scripts of text processing commands; the <I>sed</I> tool, along with <I>sed</I>
scripts, are used to deal with common APIs (for UNIX-like systems) included in
current TenDRA distributions. We think that the second method is more flexible;
for example, the text processing commands to apply on the system header files in
order to build successfully our custom APIs for Linux were in most cases the
same among Linux/i386 and Linux/Alpha targets, so we have written a single set
of sed scripts suitable for both of these platforms.</para>
<para><A HREF="linux_re.htm#TABLE1">Table 1</A> gives an
idea of the amount of work required to specify and to build the APIs used by the
~230 Linux commands which were ported to ANDF. Two types of data are reported:
`number of files' and `number of lines'. The files are modules holding lines of
either API specification or of sed commands. We excluded from our statistics the
comment lines for both, because the ratio of comments versus actual <I>tspec/sed
</I>constructs was unusually high.</para>
<para>We recall that our `Base API' for Linux was derived from
existing ``standard'' APIs, i.e. mainly from XPG3 and SVID3, which themselves
use parts of POSIX. Consequently, the number of tspec <I>lines</I> for this API
which is shown below does not reflect the actual amount of lines of
specification having been used: most of the specifications we wrote are similar
to C language ``#include'' statements (<I>tspec</I> ``+IMPLEMENT'' or ``+USE''
constructs).</para>
<para><A NAME="8557">In our `Extension API 1', we have not uncommonly rewritten
specifications which had an equivalent (either identical or similar) within the
reference SVID3 API or SPEC1070 API.</A></para>
<para><A NAME="9300">We recall also that we did not start from scratch when
making the changes in Linux system headers in order to build our APIs, either
for Linux/i386 or Linux/Alpha targets: the TenDRA Snapshot we started from
(April 95) provided 11 <I>sed</I> scripts for this purpose, that were
sufficient to build most of the XPG3 API for the Linux/i386 target.</A></para>
<para><A NAME="14204">The number of <I>files</I> given for `Changes in system
headers' is the one for the 1st target, Linux/i386 (1.0): as already mentioned,
the build of APIs for the 2nd target, Linux/Alpha (1.3), uses the same set of
sed scripts. This means that the number of <I>lines</I> corresponds to the <I>sed</I>
commands which apply either to a Linux/Intel system header, to a Linux/Alpha
header or to both.</A></para>
<para><A NAME="TABLE1"></A>
</para>
<para><B>Table 1: APIs for Linux / ~230 commands</B></para>
<informaltable>
<row>
<entry></entry>
<entry><B>Base API, specs</B></entry>
<entry><B>Extension API1, specs</B></entry>
<entry><B>Extension API2 (X11), specs</B></entry>
<entry><B>Changes in system headers (build)</B></entry></row>
<row>
<entry>files</entry>
<entry>65 (.h)</entry>
<entry>66 (.h)</entry>
<entry>3 (.h)</entry>
<entry>48 (sed scripts)</entry></row>
<row>
<entry>lines (excl. comments)</entry>
<entry>1381</entry>
<entry>684</entry>
<entry>18</entry>
<entry>254</entry></row></informaltable>
<para><A NAME="14259"></A></para>
<para><A NAME="14261"></A></para>
<para><A NAME="14262"></A></para>
<para><A NAME="14260"></A></para>
</sect1>
<sect1>
<title>6.2 Statistics concerning changes in the original source code</title>
<para><A NAME="8523"></A><A HREF="linux_re.htm#TABLE2">Table 2</A> lists the
packages we installed and validated on both platforms. The first column
gives the (Slackware Linux) name of the package, and the second gives the
number of source files we dealt with during the ANDFization. The following
column gives the number of files - actual source or Makefile - patched
during either the initial ANDFization (keeping the i386 as target), or the
port to the 2nd target (Alpha). The two last columns show the number of
files specifically (re)patched during the port to Alpha, and finally the
number of patched Makefiles (that for the overall project).</para>
<para><A NAME="TABLE2"></A>
</para>
<para><B>Table 2: Packages ported to Linux/i386 and Linux/Alpha</B></para>
<informaltable>
<row>
<entry><B>Packages</B></entry>
<entry><B>Nb source files</B></entry>
<entry><B>Nb patches <I>Total</I></B></entry>
<entry><B>Nb patches <I>Alpha port</I></B></entry>
<entry><B>Nb patches <I>Makefiles</I></B></entry></row>
<row>
<entry>bin</entry>
<entry>455 (1)</entry>
<entry>86</entry>
<entry>19</entry>
<entry>6</entry></row>
<row>
<entry>sh_util</entry>
<entry>72</entry>
<entry>14</entry>
<entry>3</entry>
<entry>0</entry></row>
<row>
<entry>txtutils</entry>
<entry>50</entry>
<entry>8</entry>
<entry>2</entry>
<entry>1</entry></row>
<row>
<entry>util</entry>
<entry>71(2)</entry>
<entry>51</entry>
<entry>22</entry>
<entry>3</entry></row>
<row>
<entry>diff</entry>
<entry>32</entry>
<entry>10</entry>
<entry>3</entry>
<entry>0</entry></row>
<row>
<entry>gzip</entry>
<entry>36</entry>
<entry>4</entry>
<entry>1</entry>
<entry>1</entry></row>
<row>
<entry>grep</entry>
<entry>16</entry>
<entry>7</entry>
<entry>4</entry>
<entry>1</entry></row>
<row>
<entry>find</entry>
<entry>57</entry>
<entry>12</entry>
<entry>2</entry>
<entry>0</entry></row>
<row>
<entry>bc</entry>
<entry>20</entry>
<entry>2</entry>
<entry>0</entry>
<entry>0</entry></row>
<row>
<entry>tar</entry>
<entry>37</entry>
<entry>10</entry>
<entry>2</entry>
<entry>1</entry></row>
<row>
<entry>rcs</entry>
<entry>28</entry>
<entry>6</entry>
<entry>0</entry>
<entry>0</entry></row>
<row>
<entry>byacc</entry>
<entry>20</entry>
<entry>8</entry>
<entry>0</entry>
<entry>0</entry></row>
<row>
<entry>m4</entry>
<entry>34</entry>
<entry>1</entry>
<entry>0</entry>
<entry>0</entry></row>
<row>
<entry>less</entry>
<entry>44</entry>
<entry>2</entry>
<entry>0</entry>
<entry>0</entry></row>
<row>
<entry>flex</entry>
<entry>46</entry>
<entry>3</entry>
<entry>0</entry>
<entry>0</entry></row>
<row>
<entry>ispell</entry>
<entry>40</entry>
<entry>21</entry>
<entry>15</entry>
<entry>1</entry></row>
<row>
<entry>elm</entry>
<entry>170</entry>
<entry>27</entry>
<entry>4</entry>
<entry>0</entry></row>
<row>
<entry>TOTAL</entry>
<entry>1233</entry>
<entry>272 (22%)</entry>
<entry>77</entry>
<entry>14</entry></row></informaltable>
<para>(1) Contents of the bin package were partially ANDFized: 48 commands out of
56. Also, a subset only of these ANDFized commands was actually ported to
Linux/Alpha: 37 (out of 48). So, the `number of source files' shown excludes
source code for non-ANDFized commands. Conversely, the `number of patches,
Alpha' would be greater if all the ANDFized commands had been ported to the
second target.</para>
<para>(2) Contents of the Slackware Linux util package were partially ANDFized: 35
commands out of 57. Among these, only 30 were actually ported to Linux/Alpha.</para>
<para><A NAME="7827"></A><A HREF="linux_re.htm#TABLE3">Table 3</A> lists the
packages we installed and validated on the Linux/i386 platform only. Similar
information, except for the number of patches for the Linux/Alpha platform, is
given.</para>
<para><A NAME="TABLE3"></A>
</para>
<para><B>Table 3: Packages ported to Linux/i386 only</B></para>
<informaltable>
<row>
<entry><B>Packages</B></entry>
<entry><B>Nb source files</B></entry>
<entry><B>Nb patches <I>(i386 only)</I></B></entry>
<entry><B>Nb patches <I>Makefiles</I></B></entry></row>
<row>
<entry>aaa_base</entry>
<entry>122</entry>
<entry>22</entry>
<entry>2</entry></row>
<row>
<entry>ash</entry>
<entry>49</entry>
<entry>15</entry>
<entry>1</entry></row>
<row>
<entry>bsdgames</entry>
<entry>47 (1)</entry>
<entry>35</entry>
<entry>11</entry></row>
<row>
<entry>cpio</entry>
<entry>44</entry>
<entry>22</entry>
<entry>1</entry></row>
<row>
<entry>getty</entry>
<entry>18</entry>
<entry>6</entry>
<entry>1</entry></row>
<row>
<entry>joe</entry>
<entry>87</entry>
<entry>26</entry>
<entry>0</entry></row>
<row>
<entry>perl</entry>
<entry>86</entry>
<entry>10</entry>
<entry>1</entry></row>
<row>
<entry>ps</entry>
<entry>35</entry>
<entry>6</entry>
<entry>1</entry></row>
<row>
<entry>sudo</entry>
<entry>9</entry>
<entry>6</entry>
<entry>0</entry></row>
<row>
<entry>tcpip/net-tools</entry>
<entry>26 (2)</entry>
<entry>12</entry>
<entry>0</entry></row>
<row>
<entry>xgames</entry>
<entry>105</entry>
<entry>28</entry>
<entry>3</entry></row>
<row>
<entry>xlock</entry>
<entry>37</entry>
<entry>4</entry>
<entry>1</entry></row>
<row>
<entry>TOTAL</entry>
<entry>665</entry>
<entry>175</entry>
<entry>22</entry></row></informaltable>
<para>(1) The bsdgames package was partially ported to ANDF</para>
<para>(2) In the tcpip package, only the Net-tools subset was ported.</para>
<para><A NAME="8168">From the data shown in </A><A HREF="linux_re.htm#TABLE2">Table
2</A>, we can say that about 80% of the original source files did not require
any change when ported to ANDF. We find also that we encountered, during the
initial ANDFization and the validation on the 1st platform, more than 70% of the
files requiring changes. In other words, we had missed approximately 30% of
required changes. Note that in our project both of the platforms were running
the same UNIX-like variant (Linux), so the last ratio could be greater in a less
favorable case. Finally, since the Intel/i386 and Digital/Alpha feature
different sizes for `pointer' type (and `long' type), while the byte ordering is
the same, there is probably still incorrect code lying inside our ANDF files for
Linux commands.</para>
<para><A NAME="6349"></A></para>
</sect1>
</chapter>
<chapter>
<title>7. Conclusion</title>
<para>During this second part of the project, we greatly benefited from
the experience acquired during the first part
<A HREF="linux_re.htm#6567">[2]</A>. The build environment we used and
the way we integrated the TenDRA technology were directly derived from
the work on the Unixware platform. Also, the initial set of Linux
commands we could reasonably port was derived from the commands we had
already ANDFized for Unixware.</para>
<para><A NAME="11487">We finally ported and validated about 230 commands
on the Linux/i386 platform, but we could port only about 150 of them to
the Linux/Alpha platform. The main reason for not porting the 80
remaining commands on Linux/Alpha was that we were short of time.
Although we are below the target of 300 commands to be ported to both
platforms, we believe that the experiment was a success. A significant
amount of sometimes complex code was ANDFized, and the portability of
this code was demonstrated. The main reasons that the initial objective
was not attained were:</A></para>
<itemizedlist>
<listitem>The switch to the second platform was delayed. Initially, we
had planned to use the microkernel based version of Linux on PowerPC,
but this was not available in time. <A NAME="11560"></A></listitem>
<listitem>Setting up the Alpha/Linux system took a considerable amount of
time. This was mainly due to the fact that Linux has not been ported to
``off-the-shelf'' hardware, and the system had to be assembled from
components (mother board, power supply, disks, etc.). </listitem>
<listitem>In the Linux/Alpha distribution we installed, a number of
commands were missing, and others were still not fully validated. This
meant that, porting commands from Intel to Alpha, we found several
portability bugs which were not fixed, and sometimes not even known, in
the Alpha command sources. Finding fixes or asking for help on the
Linux/Alpha mailing list slowed down our work.
</listitem></itemizedlist>
<para>In the course of the project, Linux has evolved from Linux 1.1 to 1.3,
and Linux commands were distributed on Alpha. However, we stuck to the
Linux 1.1 commands, and we only used the Linux/Alpha distribution to
install the system and to find fixes when we had problems with some
commands.</para>
<para>Actually, many of the portability problems we found are now fixed on
Linux/Alpha, and it would be much easier to port these commands now than
it was a few months ago. However, the first source delivery of commands
for Linux/Alpha was only available in December 95, and using this new set
of commands would have required to re-do on the Intel platform the work
(ANDFization, installation and validation) which was already done.
Moreover, this first delivery was still not extensively validated and
contained a number of bugs.</para>
<para>This project demonstrated that it is possible to build an API for a
-rather large- set of commands. Moreover, this API is mostly based on
standard APIs, xpg3, svid3, with a relatively small number of additions
(e.g. a more complete bsd API than the TenDRA bsd_extn). However,
installation of the standard APIs (Posix, Xpg3,...) with TenDRA has shown
that Linux does not fully conform to these standards. In fact, the best
support is for the bsd API, since a large number of commands come from bsd
sources.</para>
<para>We used a 3-step approach in porting to ANDF, first compiling with the
native compiler, second, using the TenDRA compiler with the native header
files, and finally using TenDRA with abstract headers files and token
libraries. Although we used the second step to find portability errors
detected by the TenDRA compiler, this could be done in the third step.
However, the third step which consists in making the source code compatible
with the abstract API, is not so easy, and thus we found it easier to
resolve portability checks in a separate step.</para>
<para>Most of the modifications we made to the source code for
the commands came from missing function declarations and other trivial
unportable coding, e.g. wrong assumptions on int and pointer sizes.</para>
<para>The TenDRA technology has proven capable of handling the differences
between the two platforms. Although the same operating system runs on both
platforms, these hardware platforms are sufficiently different to create
difficulties. Standard APIs were successfully installed on both platforms,
and the specific API developed for the commands could be designed to support
the two different implementations. However, defining and installing an API
is not an easy process and requires some -sometimes extended- knowledge of
the TenDRA technology. We also detected a very small number of bugs in the
TenDRA technology, in particular for the Dec/Alpha installer which has only
been recently distributed by DRA.</para>
</chapter>
<chapter>
<title>8. Appendix: list of problems submitted to DRA</title>
<para>The problems are classified according to their status
at the end of March 1996. They were encountered on the April 95 or
November 95 releases of TenDra, and have all been fixed now.</para>
<sect1>
<title>Errors in the TenDRA compiler</title>
<itemizedlist>
<listitem>CR95_359.FB::shift_to_sign<BR/>Error on an arithmetic shift left.<BR/>Fixed
in November 95.
</listitem>
<listitem>261 - array_sizeof<BR/>Error when declaring twice an array with sizeof(int)
as dimension.<BR/>Fixed in tdfc:4.58
<A NAME="5642"></A></listitem>
<listitem>262 - func_var<BR/>Name conflict between a function and its arguments.<BR/>Fixed
in TenDRA February 96.
<A NAME="5546"></A></listitem>
<listitem>276 - loop_tdfc_alarm<BR/>Tdfc loops when attempting to redefine as macro a
tokenized function.<BR/>Fixed in tdfc 4:58
<A NAME="5547"></A></listitem>
<listitem>220 - fstp_esi<BR/>invalid assembler instruction ``fstp %esi''<BR/>Fixed in
TenDRA November 95.
<A NAME="5548"></A></listitem>
<listitem>225 - func_return_struct<BR/>Intercallability with a gcc-compiled library
function which returns an 8-bytes structure. Bypassed with thw tcc option
-Wt,-G1.<BR/>Fixed in TenDRA November 95.
<A NAME="5552"></A></listitem>
<listitem>335 - alpha_long_too_big<BR/>Use of a literal integer constant larger than
232.<BR/>Bypassed. Added a `L' suffix to such a value.
<A NAME="5556"></A></listitem>
<listitem>337 - Error in Alpha installer<BR/>Error message ``internal error:
Change_variety out of range''.<BR/>Fixed in TenDRA February 96.
<A NAME="5566"></A></listitem>
<listitem>338 - Error in SVID3 API (RLIM_INFINITY)<BR/>Illegal type for the API
definition ``+CONST int RLIM_INFINITY ;''<BR/>Fixed in the next major TenDRA
release.
<A NAME="5567"></A></listitem>
<listitem>343 - Error in Linux installer (/tmp full)<BR/>The translation on
Linux/Alpha of a file required more than 80MB in /tmp<BR/>Fixed in the TenDRA
February 96.
<A NAME="5570"></A></listitem>
<listitem>344 - SVID3 API (daddr_t)<BR/>Casting an integer value to an unspecified
type was not rejected.<BR/>Fixed in tdfc:4.98
</listitem></itemizedlist>
</sect1>
</chapter>
<bibliography>
<title>Bibliography</title>
<biblioentry xreflabel="Ferrière95-1">
<authorgroup>
<author>
<firstname>François</firstname>
<surname>de Ferrière</surname>
</author>
<author>
<firstname>Jacques</firstname>
<surname>Febvre</surname>
</author>
</authorgroup>
<copyright>
<year>1995</year>
<holder>Open Systems Research Institute</holder>
</copyright>
<title>Revised Plan for Validation of TenDRA Capability to Implement a
Unix-like Operating System</title>
June 1995
</biblioentry>
<biblioentry xreflabel="Ferrière95-2">
<authorgroup>
<author>
<firstname>François</firstname>
<surname>de Ferrière</surname>
</author>
<author>
<firstname>Fred</firstname>
<surname>Roy</surname>
</author>
</authorgroup>
<copyright>
<year>1995</year>
<holder>Open Systems Research Institute</holder>
</copyright>
<para><A NAME="6567"></A>
<title>Validation of TenDRA Capability to Implement a UNIX-like
Operating System</title>June 1995</para>
</biblioentry>
</bibliography>
<para>Last Modified: 04:46pm , July 30, 1996</para>
</book>