Recursive Make Considered Harmful -- Building Multiple Programs

I have a number of test programs in a testsuite for an embedded system. They may share source files among them in addition to some other common files. I do not feel like managing a number of full-blown makefiles in building these programs. Besides, Peter Miller has written a very excellent paper on "Recursive Make Considered Harmful". He argued that multiple makefiles with common dependencies simply do not work.

The paper above shows how to handle building a program with dependencies that come from multiple directories. How would we then build multiple programs using a single makefile without too much boilerplate cut-and-paste? The latest GNU make 3.81 with (correct) eval support makes this very easy. Just like in the paper, I depend on GCC to generate source code dependencies and also on sed, the stream editor, to transform the dependency output to what I want. But first, below is the structure of my build directory:



The build directory contains the makefile and the other directories are "module" directories.

Similar to the paper, I put makefile include file module.mk in each module directory. The source files other modules may depend on are exported through the SRC variable. For my case, however, a module may also have one or more source files that contribute to the implementation of the main function for the module (let's call this module program module.) For this purpose, I define PROG_SRC variable for declaring these source files. An example from module.mk from my dhrystone module is given below.

SRC +=
PROG_SRC := dhrystone/dhry_1.c dhrystone/dhry_2.c

Below is another example of module.mk from my common module, which do not have a main function.

SRC += common/vec.S common/swi.S common/crt0.S \ 
       common/int-handler.c common/swi-handler.c \ 
       common/intr-write.c common/swi-write.c \ 
       common/swi-clock.c common/console.c

In my singular makefile, I have variable PROGS to declare all program modules and COMMS for other "common" modules. For example:

PROGS := simple unmapped rap fiq dhrystone large
COMMS := common

To declare the modules a program module depends on, I declare makefile variable with name like <program-module>_MK. Basically, this variable lists the module.mk files from the modules the program module depends on. An example is given below for my simple program module:

simple_MK := ../common/module.mk ../simple/module.mk

The above simply says that the simple module depends on the common module and its own. Note that the "self" module.mk must be last in the list so that its PROG_SRC variable will take effect.

Given that we have defined all of the program modules and the module.mk files each depends on, how do we go about creating the rules for building the program modules? Copy-and-paste is one option but where is the fun in that? What we need is rule building template which can be "instantiated" through eval magic. The template is given below:

define PROG_template
SRC :=
include $$($(1)_MK)
$(1)_OBJ := $$(call get_objs,$$(patsubst %, ../%,$$(PROG_SRC) $$(SRC)))

$(1): $$($(1)_OBJ)
        $$(CC) $$(LDFLAGS) -o $$@ $$^
endef

So if you were to call PROG_template with argument simple, make would have emitted this text:

SRC :=
include $(simple_MK)
simple_OBJ := ../simple/simple.o ../common/vec.S # ...and other common files

simple: $(simple_OBJ)
        $(CC) $(LDFLAGS) -o $@ $^

All module.mk files simple depends on are first included, thus filling in the SRC and PROG_SRC variables. Next, the object files it depends on are listed followed by the rule to build simple. By the way, function get_objs called above converts all .c and .S filenames into .o filenames:

define get_objs
 $(patsubst %.S,%.o,  $(filter %.S,$(1)))  $(patsubst %.c,%.o,  $(filter %.c,$(1)))
endef

Now, the trick is to "evaluate" the emitted text for all of the program modules:

$(foreach t,$(PROGS),$(eval $(call PROG_template,$(t))))

And finally, one rule to build all program modules:

progs: $(PROGS)

That is all there is to it. Below is the complete makefile for your reference.

.SUFFIXES:
.SUFFIXES: .h .c .S .o .lst .sym .d

# list the test program to build. There should be a directory for each
# test program with the same name. COMMS are for directories that contains
# common source files but do not contain main function.

PROGS := simple unmapped rap fiq dhrystone large
COMMS := common

# define the module.mk files to include for building each test. Each
# module.mk defines the source files it exports in variable SRC and,
# if the module defines a program, the program source file(s) in
# PROG_SRC. Put the module.mk for the program last.

simple_MK := ../common/module.mk ../simple/module.mk
unmapped_MK := ../common/module.mk ../unmapped/module.mk
rap_MK := ../common/module.mk ../rap/module.mk
fiq_MK := ../common/module.mk ../fiq/module.mk
dhrystone_MK := ../common/module.mk ../dhrystone/module.mk
large_MK := ../common/module.mk ../large/module.mk

# !!!DO NOT CHANGE ANYTHING BELOW THIS LINE!!!

TARGET:=arm-elf
CC:=$(TARGET)-gcc
AS:=$(TARGET)-as
LD:=$(TARGET)-ld
OBJDUMP:=$(TARGET)-objdump
NM:=$(TARGET)-nm
OBJCOPY:=$(TARGET)-objcopy
STRIP:=$(TARGET)-strip
RM:=rm

SED:=sed
LDSCRIPT:=zero.ld

XDEFINES:=-DHZ=100
XINCLUDES:=$(COMMS:%=-I../%) $(PROGS:%=-I../%)
DFLAGS:=-g
OFLAGS:=-O2 -fomit-frame-pointer
WFLAGS:=-ansi -Wall -Wstrict-prototypes -Wno-trigraphs
CFLAGS:=-mcpu=arm7tdmi $(DFLAGS) $(OFLAGS) $(WFLAGS) \ 
         $(XDEFINES) $(DEFINES) $(XINCLUDES) $(INCLUDES)
ASFLAGS := $(SDDEFINES) $(XDEFINES) $(DEFINES) $(XINCLUDES) $(INCLUDES)
LDFLAGS := $(CFLAGS) -Wl,--script=$(LDSCRIPT) -nostartfiles

#objdump flags for generating listing files
ODFLAGS :=

.PHONY: all progs listings clean clean-dep clean-all

all: progs listings

progs: $(PROGS)

listings: $(PROGS:%=%.lst) $(PROGS:%=%.sym)

# deduce object files from .S and .c files

define get_objs
 $(patsubst %.S,%.o, $(filter %.S,$(1))) $(patsubst %.c,%.o, $(filter %.c,$(1)))
endef

# an eval template for deducing object files and rule for a program.
# $(1) is the test program name.

define PROG_template
SRC :=
include $$($(1)_MK)
$(1)_OBJ := $$(call get_objs,$$(patsubst %, ../%,$$(PROG_SRC) $$(SRC)))

$(1): $$($(1)_OBJ)
 $$(CC) $$(LDFLAGS) -o $$@ $$^
endef

# generate the rule for building each of the programs

$(foreach t,$(PROGS),$(eval $(call PROG_template,$(t))))

# generate all of the object files from all test test programs
# and use it to include all of the dependency files.

ALL_OBJ :=
$(foreach t,$(PROGS),$(eval ALL_OBJ += $$($(t)_OBJ)))
ALL_OBJ := $(sort $(ALL_OBJ))

include $(ALL_OBJ:.o=.d)

clean:
        -$(RM) $(PROGS) $(PROGS:%=%.lst) $(PROGS:%=%.sym) \ 
               $(ALL_OBJ)

clean-dep:
        -$(RM) $(ALL_OBJ:.o=.d)

clean-all: clean clean-dep

%.lst: %
        $(OBJDUMP) $(ODFLAGS) -d $< > $@

%.sym: %
        $(NM) -n $< > $@

%.d: %.c
        @echo generating dependencies from $<
        @$(CC) $(CFLAGS) -MM -MG $< | \ 
        $(SED) 's%^\(.*\)\.o%$(dir $@)\1.d $(dir $@)\1.o%' > $@

%.d: %.S
        @echo generating dependencies from $<
        @$(CC) $(ASFLAGS) -MM -MG $< | \ 
        $(SED) 's%^\(.*\)\.o%$(dir $@)\1.d $(dir $@)\1.o%' > $@

(Note: You need to change the space characters before each command in a rule with a tab!)

Newlib Angel SWI handing in C

Previously, I have shown how to implement Angel SWI write command so that printf and friends would work on a UART. That implementation was rather simplistic because the UART writing loop will hold the caller from doing any other useful stuff. What I want to do now is to make use of FIFO in the UART and also interrupt it only when the FIFO is empty enough.

But first, I want to change the SWI handling from using ARM assembly code into C because the implementation now has become more sophisticated. Before you continue reading, I think you should also check the thread I started in the GNUARM mailing list on this subject where there were a lot of insights that can be gathered from the contributors to the thread.

It seems like GCC supports C function that can act as the handler to the various exceptions that ARM can throw through the __attribute__ modifier. Below is an example of a SWI handler that would accept an Angel SWI call from newlib and see if it is a write command and write the bytes in the string to a specific location.

static int *out = (int *) 0x08000000;

int __attribute__((interrupt("SWI")))
 handle_swi(int reason, void *args) {

 int i, n, *a;
 char *s;
 if (reason != 5) return -1;
 a = (int*) args;
 s = (char *) a[1];
 n = a[2];
 for (i=0; i<n; ++i) *out = s[i];
 return 0;
}

Note that for Angel SWI write to work, we need to return the number of bytes yet to be written. So zero is returned above because the function has written out all of the bytes. Now, if we link this function to the ARM vector address for SWI, it looks like it may work. But look closely at the generated assembly code for the function:

00000000 <handle_swi>:
  0:  cmp   r0, #5 ; 0x5
  4:  stmdb sp!, {r0, r1, r2, r3, ip}
  8:  mvnne r0, #0 ; 0x0
  c:  beq   18 <handle_swi+0x18>
 10:  ldmia sp!, {r0, r1, r2, r3, ip}
 14:  movs  pc, lr
 18:  ldr   r0, [r1, #8]
 1c:  cmp   r0, #0 ; 0x0
 20:  ldr   r1, [r1, #4]
 24:  ble   44 <handle_swi+0x44>
 28:  mov   r2, #0 ; 0x0
 2c:  mov   ip, #134217728 ; 0x8000000
 30:  ldrb  r3, [r2, r1]
 34:  add   r2, r2, #1 ; 0x1
 38:  cmp   r0, r2
 3c:  str   r3, [ip]
 40:  bne   30 <handle_swi+0x30>
 44:  mov   r0, #0 ; 0x0
 48:  b     10 <handle_swi+0x10>

Upon entry, r0 is among a few registers that is saved on the stack. Upon return, at addresses 0x10 and 0x14, register r0 and the other registers are restored. As such the return value, which is set at address 0x44, has been clobbered.

After some more reading on GCC function attributes, it appeared that attribute naked could be used, but we need to insert inline assembly code for the prologue and epilogue ourselves. Somebody in the GNUARM mailing list thread also suggested something along this line. That is, we would do something like this instead:

static int *out = (int *) 0x08000000;

int __attribute__((naked))
 handle_swi(int reason, void *args) {

 asm("stmdb sp!,{r4-r11,ip,lr}");
 int r, i, n, *a;
 char *s;
 r = 0;
 if (reason != 5) r = -1;
 else {
  a = (int*) args;
  s = (char *) a[1];
  n = a[2];
  for (i=0; i<n; ++i) *out = s[i];
 }
 asm("ldmia sp!,{r4-r11,ip,lr};movs pc,lr");
 return r;
}

So much for trying to avoid assembly code in SWI handler. I choose not to save r0 to r3 because they are allowed to be clobbered in a function called. But the rest of the registers need to be saved and restored because we won't know how the C code would generally use them.

At this point, I came to the conclusion that avoiding assembly code from SWI handler is rather difficult. Actually, we need to do more than just the epilogue and prologue code. So instead of sprinkling inline assembly code in the C function, I came up with a consolidated assembly code wrapper that does the necessary checking and other preparation before calling a straight C function to handle the SWI call.

__swi_handler:
  stmdb  sp!,{r4,lr}

  /* see if SWI argument is 0x123456 */
  ldr    r4,[lr,#-4]
  bic    r4,r4,#0xff000000
  sub    r4,r4,#0x00120000
  sub    r4,r4,#0x00003400
  subs   r4,r4,#0x00000056

  /* save SPSR so that we have SWI reentrancy */
  mrs    r4,spsr
  stmdb  sp!,{r4}

  /* only call handler if SWI argument is 0x123456 */
  bleq   swi_handler

  /* restore SPSR */  
  ldmia  sp!,{r4}
  msr    spsr,r4

  ldmia  sp!,{r4,pc}^

Now we just need to make sure SWI vector address calls __swi_handler. And remove the attribute modifier from swi_handler function definition.

Actually, there are more to SWI handling than just Angel SWI as can be gathered from the GNUARM thread mentioned earlier. For example, someone pointed out that divide-by-zero error will cause a SWI call. We may want to handle this properly too.

Mind your struct's!

This should be elementary to seasoned C programmers but one rule of thumb is, in a struct, one should arrange the fields based on the size of their types in reverse order. This would typically lead to a more compact struct which can be crucial to an embedded system with limited memory. To illustrate, struct_a below is a struct similar to what I've found in a library provided by a partner of my company. One the other hand, struct_b is the same struct but with its fields "properly ordered". In addition, the array arr_a also appears in similar fashion in the partner's library.

#include 

struct struct_a {
  short f1;
  char  f2;
  int   f3;
  char  f4;
};

struct struct_b {
  int f3;
  short f1;
  char f2;
  char f4;
};

struct struct_a arr_a[32];
struct struct_b arr_b[32];

int main() {
  printf("sizeof(struct_a):%d\n", sizeof(struct struct_a));
  printf("sizeof(struct_b):%d\n", sizeof(struct struct_b));
  printf("   sizeof(arr_a):%d\n", sizeof(arr_a));
  printf("   sizeof(arr_b):%d\n", sizeof(arr_b));
  return 0;
}

Compiling the code above using the GNUARM toolchain and running it the ARM simulator in GDB, we get this output:

sizeof(struct_a):12
sizeof(struct_b):8
   sizeof(arr_a):384
   sizeof(arr_b):256

Simply by reordering the fields, we could save 4 bytes in the struct and 128 bytes from the array above. Another example is given below. Again struct_a is something found in the library and struct_b is an optimized version of the struct.

#include 

struct struct_a {
   short f1;
   int   f2;
   char  f3;
   char  f4;
   short f5;
   char  f6;
   short f7;
   short f8[3];
};

struct struct_b {
   int   f2;
   short f8[3];
   short f1;
   short f7;
   short f5;
   char  f3;
   char  f4;
   char  f6;
};

struct struct_a arr_a[32];
struct struct_b arr_b[32];

int main() {
  printf("sizeof(struct_a):%d\n", sizeof(struct struct_a));
  printf("sizeof(struct_b):%d\n", sizeof(struct struct_b));
  printf("   sizeof(arr_a):%d\n", sizeof(arr_a));
  printf("   sizeof(arr_b):%d\n", sizeof(arr_b));
  return 0;
}

The output of the program compiled through GNUARM toolchain is given below.

sizeof(struct_a):24
sizeof(struct_b):20
   sizeof(arr_a):768
   sizeof(arr_b):640

Again, we could save 4 bytes from the struct and 128 bytes from the array. Note that struct_b above has a short array with 3 elements as a field. Why it was not made the first field of struct_b? After all it is bigger than an int. It has something to do with int alignment on ARM. Try it out and see.

Using interrupts for super responsive tasks?

I read an interesting article at Embedded.com titled "A data-centric OS for MCUs using a real-time publisher-subscriber-mechanism: Part1" and it is very interesting. Typically, in embedded system, you would use a preemptive OS which run tasks based on task priorities on a predefined timeslice. This article, however, proposes a publish-and-subscribe task model using interrupts and interrupt priorities. So no OS in the traditional sense. Basically, each task is an interrupt handler.

A task can be a consumer or a producer. The producer task would trigger an interrupt through software when an interesting event occurs. The consumer task would subscribe to that event by registering an interrupt handler as the event call back function. The interrupt priorities would be used to decide which event handler would be called in the event that many event handlers are eligible to run. Nested interrupt is also enabled.

The interrupt controller we are using for our SoC supports up to 64 interrupt sources that can also be programmatically triggered. It also has 16 interrupt priorities with programmable interrupt handler each to play with. It would be interesting to see how the suggested technique in the article could be mapped to it.

But it worked before...

I helped a colleague early last week. His PPP server running on VxWorks on our Carmen II board used to run okay a few weeks back. But when he started to work on it again a couple of weeks ago, it gave no response to PPP connection attempts from a Windows PC. My first instinct was to check the RS232 connection to the board because the connection requires manual wiring to the board (the only DB-9 connector on the board has been allocated for the console). However, according to my colleague the connection worked because his AT command processor running on the board received bytes from the PC. Moreover, the test program we had to test the two serial ports seemed to transfer bytes just fine. So both of us went on to debug the PPP configuration. We went as far as using the base BSP we have by including PPP in it. Still we could not get PPP to work. The following morning, I finally decided to look at the connection. Well, lo and behold, the connection was "half correct" which explains why the AT command processor could still receive bytes. I think you can guess what had happened and I will leave it at that.

Yesterday, I helped another colleague to debug the board with the design she had downloaded into the FPGA. A "similar" design had worked before. By "working", we mean we could get our in-circuit-emulator (ICE) to bring ARM into the background debug mode (BDM). She has been debugging her design for a number of days already. Initially, I thought the ICE used the DBGRQ and DBGACK signals to enter BDM. Apparently not. So there is no other way for the ICE to do that except through the JTAG serial protocol. With the help of a logic analyzer we watched the 5 JTAG signals both with the good and the bad designs on the board's FPGA. Immediately we saw that there was some problem with the nTRST signal. The signal was oscillating on the bad design whereas, in the good design, it started low and then high, followed by some activity on the TMS, TDI and TDO signals. Looking at the board schematic, we could see that the nTRST signal from the JTAG connector goes first to the FPGA and then from there to the ARM. So there must have been bad connection between these two points on the FPGA. Sure enough. When she rechecked her UCF file for the design, the nTRST input into the FPGA was not connected to the reset controller module which drives some other logic before going out to the ARM.

So, the lesson is, I guess, before going on debugging your design or code check all of the manual connections you have made. Especially when it has worked before.

newlib printf through UART on GNU ARM toolchain

To make function like printf and puts to work on the GNUARM toolchain, the Angel SWI handler for the AngelSWI_Reason_Write command needs to be implemented. This command is indicated through value 5 in register r0 when the Angel SWI call is made. Register r1 will point to an array of 3 32-bit values. The first is a file pointer which I will ignore because I do not do any other file operation except writing to stdout. The second is a pointer to the array of char's to print. The third contains the length of the string. Below is the code snippet for the SWI handler:

__swi_handler:
  cmp    r0,#5
  bne    .Ldone

  /* skip file pointer and store char pointer in r0
     and length into r1 */

  ldr    r0,[r1,#4]!
  ldr    r1,[r1,#4]

  add    r1,r1,r0  /* now r1 points to end of chars to print */

  mov    r2,#UARTA_BASE

.Lnext:
  ldr    r3,[r2,#LSR]
  ands   r3,r3,#LSR_THRE
  beq    .Lnext
  
  ldrb   r3,[r0],#1
  str    r3,[r2,#THR]

  /* append CR if we see a NL */

  cmp    r3,#0x0A    /* NL? */
  moveq  r3,#0x0D    /* CR  */
  streq  r3,[r2,#THR]

  cmp    r0,r1
  blt    .Lnext
  mov    r0,#0
.Ldone:
  movs  pc,lr

I am being a little bit sloppy above by not checking that the SWI instruction that triggers the call does in fact have a 0x123456 as its argument which is required for Angel SWI. Of course, the ARM exception vectors needs to be configured accordingly:

__vec_start__:
  LDR   PC, Reset_Addr   
  LDR   PC, Undef_Addr
  LDR   PC, SWI_Addr
  LDR   PC, PAbt_Addr
  LDR   PC, DAbt_Addr
  NOP
  LDR   PC, IRQ_Addr
  LDR   PC, FIQ_Addr

Reset_Addr:   .word   start
Undef_Addr:   .word   Undef_Handler
SWI_Addr:     .word   __swi_handler
PAbt_Addr:    .word   PAbt_Handler
DAbt_Addr:    .word   DAbt_Handler
             .word   0
IRQ_Addr:     .word   IRQ_Handler
FIQ_Addr:     .word   FIQ_Handler

And the UART needs to be configured properly before use. I have the code below in my crt0.S to initialize the UART. You need to program the DLL and DLH (Divisor Latch registers) according to your UART clock and the desired baud rate:

  mov    r0,#UARTA_BASE
  mov    r1,#0x07
  str    r1,[r0,#FCR]  /* reset and enable FIFOs */
  mov    r1,#0x00
  str    r1,[r0,#IER]  /* no interrupt */
  mov    r1,#0x80
  str    r1,[r0,#LCR]  /* to program divisor */
  mov    r1,#0x41      /* change this if clock change! */
  str    r1,[r0,#DLL]
  mov    r1,#0x00
  str    r1,[r0,#DLH]
  mov    r1,#0x03
  str    r1,[r0,#LCR]  /* 8 bit, 1 stop bit */

Java SE for Embedded Use

I supposed Sun has now realized that there is a place for Java SE (Java Platform, Standard Edition), in addition to Java ME (Java Platform, Mobile Edition), in the embedded market. They have just released an early access to Java SE for PowerPC running Linux. See http://java.sun.com/j2se/embedded/. But to download you need to answer a questionnaire first. I guess it is good to complete the questionnaire to give Sun what embedded platforms we would like to see Java SE runs on. (But the questionnaire does not list my country for me to pick!)

We are currently using ARM7 at work. Since the full-fledge Linux does not run on ARM7 due to a missing MMU, I guess we won't be seeing Java SE for ARM7 any time soon. Unless someone does the porting to uClinux.

It's hard to pick a name!

It is hard to pick a good name for my blog!

I have been doing embedded software development on ARM processor at my new job for about a year now. I thought kARMa would be a good name play with ARM moniker. Only that name is taken already. So I have to settle with "Koda.Karma."

I would like to believe "koda" is a corruption of the English word "code" into the Malay language but I could not confirm this from the scarce Malay dictionaries on the Internet.

So there you have it. My first blog entry here and an attempt at a good sounding blog name. Welcome to Koda.Karma!