Subversion Repositories tendra.SVN

Rev

Rev 2 | Details | Compare with Previous | Last modification | View Log | RSS feed

Rev Author Line No. Line
2 7u83 1
<!-- Crown Copyright (c) 1998 -->
2
<HTML>
3
<HEAD>
4
<TITLE>
5
C++ Producer Guide: Token syntax 
6
</TITLE>
7
</HEAD>
8
<BODY TEXT="#000000" BGCOLOR="#FFFFFF" LINK="#0000FF" VLINK="#400080" ALINK="#FF0000">
9
 
10
<H1>C++ Producer Guide</H1>
11
<H3>March 1998</H3>
12
<A HREF="dump.html"><IMG SRC="../images/next.gif" ALT="next section"></A>
13
<A HREF="pragma.html"><IMG SRC="../images/prev.gif" ALT="previous section"></A>
14
<A HREF="index.html"><IMG SRC="../images/top.gif" ALT="current document"></A>
15
<A HREF="../index.html"><IMG SRC="../images/home.gif" ALT="TenDRA home page">
16
</A>
17
<IMG SRC="../images/no_index.gif" ALT="document index"><P>
18
<HR>
19
 
20
<DL>
21
<DT><A HREF="#spec"><B>2.3.1</B> - Token specifications</A><DD>
22
<DT><A HREF="#args"><B>2.3.2</B> - Token arguments</A><DD>
23
<DT><A HREF="#tokdef"><B>2.3.3</B> - Defining tokens</A><DD>
24
</DL>
25
<HR>
26
 
27
<H2>2.3. Token syntax</H2>
28
<P>
29
The C and C++ producers allow place-holders for various categories
30
of syntactic classes to be expressed using directives of the form:
31
<PRE>
32
	#pragma TenDRA token <I>token-spec</I>
33
</PRE>
34
or simply: 
35
<PRE>
36
	#pragma token <I>token-spec</I>
37
</PRE>
38
These place-holders are represented as TDF tokens and hence are called
39
tokens.  These tokens stand for a certain type, expression or whatever
40
which is to be represented by a certain named TDF token in the producer
41
output.  This mechanism is used, for example, to allow C API specifications
42
to be represented target independently.  The types, functions and
43
expressions comprising the API can be described using <CODE>#pragma
44
token</CODE> directives and the target dependent definitions of these
45
tokens, representing the implementation of the API on a particular
46
machine, can be linked in later.  This mechanism is described in detail
47
elsewhere. 
48
</P>
49
<P>
50
A <A HREF="pragma1.html#token">summary of the grammar</A> for the
51
<CODE>#pragma token</CODE> directives accepted by the C++ producer
52
is given as an annex. 
53
</P>
54
 
55
<HR>
56
<H3><A NAME="spec">2.3.1. Token specifications</A></H3>
57
<P>
58
A token specification is divided into two components, a 
59
<I>token-introduction</I> giving the token sort, and a 
60
<I>token-identification</I> giving the internal and external token
61
names: 
62
<PRE>
63
	<I>token-spec</I> :
64
		<I>token-introduction token-identification</I>
65
 
66
	<I>token-introduction</I> :
67
		<I>exp-token</I>
68
		<I>statement-token</I>
69
		<I>type-token</I>
70
		<I>member-token</I>
71
		<I>procedure-token</I>
72
 
73
	<I>token-identification</I> :
74
		<I>token-namespace<SUB>opt</SUB> identifier</I> # <I>external-identifier<SUB>opt</SUB></I>
75
 
76
	<I>token-namespace</I> :
77
		TAG
78
 
79
	<I>external-identifier</I> :
80
		-
81
		<I>preproc-token-list</I>
82
</PRE>
83
The <CODE>TAG</CODE> qualifier is used to indicate that the internal
84
name lies in the C tag namespace.  This only makes sense for structure
85
and union types.  The external token name can be given by any sequence
86
of preprocessing tokens.  These tokens are not macro expanded.  If
87
no external name is given then the internal name is used.  The special
88
external name <CODE>-</CODE> is used to indicate that the token does
89
not have an associated external name, and hence is local to the current
90
translation unit.  Such a local token must be defined.  White space
91
in the external name (other than at the start or end) is used to indicate
92
that a TDF unique name should be used.  The white space serves as
93
a separator for the unique name components. 
94
</P>
95
 
96
<H4><A NAME="exp">Expression tokens</A></H4>
97
<P>
98
Expression tokens are specified as follows: 
99
<PRE>
100
	<I>exp-token</I> :
101
		EXP <I>exp-storage<SUB>opt</SUB></I> : <I>type-id</I> :
102
		NAT
103
		INTEGER
104
</PRE>
105
representing a expression of the given type, a non-negative integer
106
constant and general integer constant, respectively.  Each expression
107
has an associated storage class: 
108
<PRE>
109
	<I>exp-storage</I> :
110
		lvalue
111
		rvalue
112
		const
113
</PRE>
114
indicating whether it is an lvalue, an rvalue or a compile-time constant
115
expression.  An absent <I>exp-storage</I> is equivalent to 
116
<CODE>rvalue</CODE>.  All expression tokens lie in the macro namespace;
117
that is, they may potentially be defined as macros. 
118
</P>
119
<P>
120
For backwards compatibility with the C producer, the directive:
121
<PRE>
122
	#pragma TenDRA++ rvalue token as const <I>allow</I>
123
</PRE>
124
causes <CODE>rvalue</CODE> tokens to be treated as <CODE>const</CODE>
125
tokens.
126
 
127
<H4>Statement tokens</H4>
128
<P>
129
Statement tokens are specified as follows: 
130
<PRE>
131
	<I>statement-token</I> :
132
		STATEMENT
133
</PRE>
134
All statement tokens lie in the macro namespace. 
135
</P>
136
 
137
<H4>Type tokens</H4>
138
<P>
139
Type tokens are specified as follows: 
140
<PRE>
141
	<I>type-token</I> :
142
		TYPE
143
		VARIETY
144
		VARIETY signed
145
		VARIETY unsigned
146
		FLOAT
147
		ARITHMETIC
148
		SCALAR
149
		CLASS
150
		STRUCT
151
		UNION
152
</PRE>
153
representing a generic type, an integral type, a signed integral type,
154
an unsigned integral type, a floating point type, an arithmetic (integral
155
or floating point) type, a scalar (arithmetic or pointer) type, a
156
class type, a structure type and a union type respectively. 
157
</P>
158
<P>
159
<IMG SRC="../images/warn.gif" ALT="warning">
160
Floating-point, arithmetic and scalar token types have not yet been
161
implemented correctly in either the C or C++ producers. 
162
</P>
163
 
164
<H4><A NAME="member">Member tokens</A></H4>
165
<P>
166
Member tokens are specified as follows: 
167
<PRE>
168
	<I>member-token</I> :
169
		MEMBER <I>access-specifier<SUB>opt</SUB> member-type-id</I> : <I>type-id</I> :
170
</PRE>
171
where an <I>access-specifier</I> of <CODE>public</CODE> is assumed
172
if none is given.  The member type is given by: 
173
<PRE>
174
	<I>member-type-id</I> :
175
		<I>type-id</I>
176
		<I>type-id</I> % <I>constant-expression</I>
177
</PRE>
178
where <CODE>%</CODE> is used to denote bitfield members (since 
179
<CODE>:</CODE> is used as a separator).  The second type denotes the
180
structure or union the given member belongs to.  Different types can
181
have members with the same internal name, but the external token name
182
must be unique.  Note that only non-static data members can be represented
183
in this form. 
184
</P>
185
<P>
186
Two declarations for the same <CODE>MEMBER</CODE> token (including token
187
definitions) should have the same type, however the directive:
188
<PRE>
189
	#pragma TenDRA++ incompatible member declaration <I>allow</I>
190
</PRE>
191
allows declarations with different types, provided these types have the
192
same size and alignment requirements.
193
</P>
194
 
195
<H4>Procedure tokens</H4>
196
<P>
197
Procedure, or high-level, tokens are specified in one of three ways:
198
<PRE>
199
	<I>procedure-token</I> :
200
		<I>general-procedure</I>
201
		<I>simple-procedure</I>
202
		<I>function-procedure</I>
203
</PRE>
204
All procedure tokens (except ellipsis functions - see below) lie in
205
the macro namespace.  The most general form of procedure token specifies
206
two sets of parameters.  The bound parameters are those which are
207
used in encoding the actual TDF output, and the program parameters
208
are those which are <A HREF="#args">specified in the program</A>.
209
The program parameters are expressed in terms of the bound parameters.
210
A program parameter can be an expression token parameter, a statement
211
token parameter, a member token parameter, a procedure token parameter
212
or any type.  The bound parameters are deduced from the program parameters
213
by a similar process to that used in template argument deduction.
214
<PRE>
215
	<I>general-procedure</I> :
216
		PROC { <I>bound-toks<SUB>opt</SUB></I> | <I>prog-pars<SUB>opt</SUB></I> } <I>token-introduction
217
</I>
218
 
219
	<I>bound-toks</I> :
220
		<I>bound-token</I>
221
		<I>bound-token</I> , <I>bound-toks</I>
222
 
223
	<I>bound-token</I> :
224
		<I>token-introduction token-namespace<SUB>opt</SUB> identifier</I>
225
 
226
	<I>prog-pars</I> :
227
		<I>program-parameter</I>
228
		<I>program-parameter</I> , <I>prog-pars</I>
229
 
230
	<I>program-parameter</I> :
231
		EXP <I>identifier</I>
232
		STATEMENT <I>identifier</I>
233
		TYPE <I>type-id</I>
234
		MEMBER <I>type-id</I> : <I>identifier</I>
235
		PROC <I>identifier</I>
236
</PRE>
237
</P>
238
<P>
239
The simplest form of a <I>general-procedure</I> is one in which the
240
<I>prog-pars</I> correspond precisely to the <I>bound-toks</I>.  In
241
this case the syntax: 
242
<PRE>
243
	<I>simple-procedure</I> :
244
		PROC ( <I>simple-toks<SUB>opt</SUB></I> ) <I>token-introduction</I>
245
 
246
	<I>simple-toks</I> :
247
		<I>simple-token</I>
248
		<I>simple-token</I> , <I>simple-toks</I>
249
 
250
	<I>simple-token</I> :
251
		<I>token-introduction token-namespace<SUB>opt</SUB> identifier<SUB>opt</SUB></I>
252
</PRE>
253
may be used.  Note that the parameter names are optional. 
254
</P>
255
<P>
256
A function token is specified as follows: 
257
<PRE>
258
	<I>function-procedure</I> :
259
		FUNC <I>type-id</I> :
260
</PRE>
261
where the given type is a function type.  This has two effects: firstly
262
a function with the given type is declared; secondly, if the function
263
type has the form: 
264
<PRE>
265
	r ( p1, ...., pn )
266
</PRE>
267
a procedure token with sort: 
268
<PRE>
269
	PROC ( EXP rvalue : p1 :, ...., EXP rvalue : pn : ) EXP rvalue : r :
270
</PRE>
271
is declared.  For ellipsis function types only the function, not the
272
token, is declared.  Note that the token behaves like a macro definition
273
of the corresponding function.  Unless explicitly enclosed in a linkage
274
specification, a function declared using a <CODE>FUNC</CODE>
275
token has C linkage.  Note that it is possible for two <CODE>FUNC</CODE>
276
tokens to have the same internal name, because of function overloading,
277
however external names must be unique. 
278
</P>
279
<P>
280
The directive: 
281
<PRE>
282
	#pragma TenDRA incompatible interface declaration <I>allow</I>
283
</PRE>
284
can be used to allow incompatible redeclarations of functions declared
285
using <CODE>FUNC</CODE> tokens.  The token declaration takes precedence.
286
</P>
287
<P>
288
<IMG SRC="../images/warn.gif" ALT="warning">
289
Certain of the more complex examples of <CODE>PROC</CODE> tokens such
290
as, for example, tokens with <CODE>PROC</CODE> parameters, have not
291
been implemented in either the C or C++ producers. 
292
</P>
293
 
294
<HR>
295
<H3><A NAME="args">2.3.2. Token arguments</A></H3>
296
<P>
297
As mentioned above, the program parameters for a <CODE>PROC</CODE>
298
token are those specified in the program itself.  These arguments
299
are expressed as a comma-separated list enclosed in brackets, the
300
form of each argument being determined by the corresponding program
301
parameter. 
302
</P>
303
<P>
304
An <CODE>EXP</CODE> argument is an assignment expression.  This must
305
be an lvalue for <CODE>lvalue</CODE> tokens and a constant expression
306
for 
307
<CODE>const</CODE> tokens.  The argument is converted to the token
308
type (for <CODE>lvalue</CODE> tokens this is essentially a conversion
309
between the corresponding reference types).  A <CODE>NAT</CODE> or
310
<CODE>INTEGER</CODE> argument is an integer constant expression. 
311
In the former case this must be non-negative. 
312
</P>
313
<P>
314
A <CODE>STATEMENT</CODE> argument is a statement.  This statement
315
should not contain any labels or any <CODE>goto</CODE> or <CODE>return</CODE>
316
statements. 
317
</P>
318
<P>
319
A type argument is a type identifier.  This must name a type of the
320
correct category for the corresponding token.  For example, a 
321
<CODE>VARIETY</CODE> token requires an integral type. 
322
</P>
323
<P>
324
<A NAME="offset">A member argument must describe the offset of a member
325
or nested member of the given structure or union type</A>.  The type
326
of the member should agree with that of the <CODE>MEMBER</CODE> token.
327
The general form of a member offset can be described in terms of member
328
selectors and array indexes as follows: 
329
<PRE>
330
	<I>member-offset</I> :
331
		::<I><SUB>opt</SUB> id-expression</I>
332
		<I>member-offset</I> . ::<I><SUB>opt</SUB> id-expression</I>
333
		<I>member-offset</I> [ <I>constant-expression</I> ]
334
</PRE>
335
</P>
336
<P>
337
A <CODE>PROC</CODE> argument is an identifier.  This identifier must
338
name a <CODE>PROC</CODE> token of the appropriate sort. 
339
</P>
340
 
341
<HR>
342
<H3><A NAME="tokdef">2.3.3. Defining tokens</A></H3>
343
<P>
344
Given a token specification of a syntactic object and a normal language
345
definition of the same object (including macro definitions if the
346
token lies in the macro namespace), the producers attempt to unify
347
the two by defining the TDF token in terms of the given definition.
348
Whether the token specification occurs before or after the language
349
definition is immaterial.  Unification also takes place in situations
350
where, for example, two types are known to be compatible.  Multiple
351
consistent explicit token definitions are allowed by default when
352
allowed by the language; this is controlled by the directive: 
353
<PRE>
354
	#pragma TenDRA compatible token <I>allow</I>
355
</PRE>
356
The default unification behaviour may be modified using the directives:
357
<PRE>
358
	#pragma TenDRA no_def <I>token-list</I>
359
	#pragma TenDRA define <I>token-list</I>
360
	#pragma TenDRA reject <I>token-list</I>
361
</PRE>
362
or equivalently: 
363
<PRE>
364
	#pragma no_def <I>token-list</I>
365
	#pragma define <I>token-list</I>
366
	#pragma ignore <I>token-list</I>
367
</PRE>
368
which set the state of the tokens given in <I>token-list</I>.  A state
369
of <CODE>no_def</CODE> means that no unification is attempted and
370
that any attempt to explicitly define the token results in an error.
371
A state of <CODE>define</CODE> means that unification takes place
372
and that the token must be defined somewhere in the translation unit.
373
A state of <CODE>reject</CODE> means that unification takes place as
374
normal, but any resulting token definition is discarded and not output
375
to the TDF capsule. 
376
</P>
377
<P>
378
If a token with the state <CODE>define</CODE> is not defined, then the
379
behaviour depends on the sort of the token.  A <CODE>FUNC</CODE> token
380
is implicitly defined in terms of its underlying function, such as:
381
<PRE>
382
	#define f( a1, ...., an )	( f ) ( a1, ...., an )
383
</PRE>
384
Other undefined tokens cause an error.  This behaviour can be modified
385
using the directives:
386
<PRE>
387
	#pragma TenDRA++ implicit token definition <I>allow</I>
388
	#pragma TenDRA++ no token definition <I>allow</I>
389
</PRE>
390
respectively.
391
<P>
392
The primitive operations, <CODE>no_def</CODE>, <CODE>define</CODE> and
393
<CODE>reject</CODE>, can also be expressed using the context sensitive
394
directive: 
395
<PRE>
396
	#pragma TenDRA interface <I>token-list</I>
397
</PRE>
398
or equivalently: 
399
<PRE>
400
	#pragma interface <I>token-list</I>
401
</PRE>
402
By default this is equivalent to <CODE>no_def</CODE>, but may be modified
403
by inclusion using one of the directives: 
404
<PRE>
405
	#pragma TenDRA extend <I>header-name</I>
406
	#pragma TenDRA implement <I>header-name</I>
407
</PRE>
408
or equivalently: 
409
<PRE>
410
	#pragma extend interface <I>header-name</I>
411
	#pragma implement interface <I>header-name</I>
412
</PRE>
413
These are equivalent to: 
414
<PRE>
415
	#include <I>header-name</I>
416
</PRE>
417
except that the form <CODE>[....]</CODE> is allowed as a header name.
418
This is equivalent to <CODE>&lt;....&gt;</CODE> except that it starts
419
the directory search after the point at which the including file was
420
found, rather than at the start of the path (i.e. it is equivalent
421
to the 
422
<CODE>#include_next</CODE> directive found in some preprocessors).
423
The effect of the <CODE>extend</CODE> directive on the state of the
424
<CODE>interface</CODE> directive is as follows: 
425
<PRE>
426
	no_def -&gt; no_def
427
	define -&gt; reject
428
	reject -&gt; reject
429
</PRE>
430
The effect of the <CODE>implement</CODE> directive is as follows:
431
<PRE>
432
	no_def -&gt; define
433
	define -&gt; define
434
	reject -&gt; reject
435
</PRE>
436
That is to say, a <CODE>implement</CODE> directive will cause all
437
the tokens in the given header to be defined and their definitions
438
output. Any tokens included in this header by <CODE>extend</CODE>
439
may be defined, but their definitions will not be output.  This is
440
precisely the behaviour which is required to ensure that each token
441
is defined exactly once in an API library build. 
442
</P>
443
<P>
444
The lists of tokens in the directives above are expressed in the form:
445
<PRE>
446
	<I>token-list</I> :
447
		<I>token-id token-list<SUB>opt</SUB></I>
448
		# <I>preproc-token-list</I>
449
</PRE>
450
where a <I>token-id</I> represents an internal token name: 
451
<PRE>
452
	<I>token-id</I> :
453
		<I>token-namespace<SUB>opt</SUB> identifier</I>
454
		<I>type-id</I> . <I>identifier</I>
455
</PRE>
456
Note that member tokens are specified by means of both the member
457
name and its parent type.  In this type specifier, <CODE>TAG</CODE>,
458
rather than 
459
<CODE>class</CODE>, <CODE>struct</CODE> or <CODE>union</CODE>, may
460
be used in elaborated type specifiers for structure and union tokens.
461
If the 
462
<I>token-id</I> names an overloaded function then the directive is
463
applied to all <CODE>FUNC</CODE> tokens of that name.  It is possible
464
to  be more selective using the <CODE>#</CODE> form which allows the
465
external token name to be specified.  Such an entry must be the last
466
in a <I>token-list</I>. 
467
</P>
468
<P>
469
A related directive has the form: 
470
<PRE>
471
	#pragma TenDRA++ undef token <I>token-list</I>
472
</PRE>
473
which undefines all the given tokens so that they are no longer visible.
474
</P>
475
<P>
476
As noted above, a macro is only considered as a token definition if
477
the token lies in the macro namespace.  Tokens which are not in the
478
macro namespace, such as types and members, cannot be defined using
479
macros. Occasionally API implementations do define member selector
480
as macros in terms of other member selectors.  Such a token needs
481
to be explicitly defined using a directive of the form: 
482
<PRE>
483
	#pragma TenDRA member definition <I>type-id</I> : <I>identifier member-offset
484
</I>
485
</PRE>
486
where <I>member-offset</I> is <A HREF="#offset">as above</A>. 
487
</P>
488
 
489
<HR>
490
<P><I>Part of the <A HREF="../index.html">TenDRA Web</A>.<BR>Crown
491
Copyright &copy; 1998.</I></P>
492
</BODY>
493
</HTML>