GSoC Status Report #2

by sprkv5

The file /docs/internals/porting-HOWTO.txt is a rough guide to porting Valgrind to a new architecture, or a new operating system. Let’s analyze the relevance of those recommendations with my project.

—————————————————————————–
Porting Valgrind to a new OS
—————————————————————————–
This implies both a new OS, and a new platform.

1 Create necessary subdirs (copy linux/ and x86-linux/ subdirs).

2 Update configure.in (use linux and x86-linux as a guide).

3 Implement all the necessary OS-specific and platform-specific types,
functions, and macros… use the following as templates:

include/linux/core_os.h
include/x86-linux/core_platform.h
coregrind/linux/core_os.h
coregrind/x86-linux/core_platform.h

4 You’ll need to copy appropriate kernel types into vki*.h.
You’ll have to ensure that everywhere that vki_*/VKI_* types and constants
are used, that they are suitable for your new OS, otherwise factor their
usage out somehow. This will be painful.

5 In particular, you’ll need to implement the VGA_(syscall_table). You may be
able to reuse some of the generic (eg. POSIX) syscall wrappers, if the types
match. Otherwise, you’ll have to write your own new wrappers. Do this
incrementally, as system calls are hit, otherwise you’ll go crazy.

6 Probably lots more things; this list should be added to!

—————————————————————————–
Comments:
—————————————————————————–
1 The subdirs host the tests which serve a basic purpose. They determine proper functioning of the implemented types, constants and functions in Valgrind. There are two variations – ones that trigger all the syscalls (traps in our case) as in the file /memcheck/tests/x86-linux/scalar.c just to bring up the unimplemented or misbehaving syscalls, and ones that perform specific task using implemented functions, types and constants as in /memcheck/tests/linux/lsframe1.c where the purpose is to demonstrate Memcheck correctly handling a 64M array on the stack.
Since the binary is on the verge of it’s initial build, this step has not been initiated.

2 Extension of 1.

3 This step is partly misleading as the mentioned files do not exist. Implementation location of “OS-specific and platform-specific types, functions, and macros…” have changed in recent versions of Valgrind. Now they lie across various headers in /include and /coregrind. The ones that could not be implemented are plugged assertions like vg_assert(0). This is rampant in /coregrind headers.

4 This painful step is one that needs one to figure out each constant’s type and declarations as they hide nested behind multiple and typedefs and #defines across many headers. Initially I waited for each of the constant to nag the the compiler with errors and then sought to find the specific code declarations. But I sought the help of three things: the implementation of other OS’s constants, the grep utility and the program Source Navigator.

The first helped me figure out the exact constants that needed to be implemented and the files in which they could be found. Often, the headers would be different because each OS has varying implementations.

The second helped me locate the constants across the numerous headers lying in /usr/include of the GNU Hurd system. The grep -rnw “path” -e “expression” command listed all the occurrences of the expression across different files with the line number.

The third – a source code browser, helped me find the implementation and types of constants and functions especially in cases of nesting.

5 This step lies just after the binary is built. I already have set up a preliminary version of the syscall table(vki-scnums-x86-gnu.h), although it’s untested and its implementation would change/evolve.

6 Yes, and the list(for Valgrind-Hurd) is another section below. See TODO.

—————————————————————————–
Porting Valgrind to a new architecture
—————————————————————————–
Note that this implies both a new architecture, and a new platform (ie. arch/OS
combination). Please add to this list if it is missing anything.

To begin:

1 Create necessary subdirs (copy x86/ and x86-linux/ subdirs). Yes, even the
test subdirectories. (You should make arch-specific tests later to be
thorough!) Put in Makefile.am files for them, edit them for the new
architecture name.

2 Update configure.in (use x86 and x86-linux as a guide).

Once it configures ok, get it to compile:

3 Create the Vex guest state type, VexGuestState

4 Copy all the arch-specific and platform-specific files into the new
directories, eg. from x86 and x86-linux. Files like (this list is not
exhaustive!):

include/x86/core_arch.h
include/x86-linux/core_platform.h
coregrind/x86/core_arch.h
coregrind/x86-linux/core_platform.h
coregrind/x86-linux/vki_arch.h
coregrind/x86-linux/vki_arch_posixtypes.h

Edit obvious things like the file headers, and the #ifdef __X86_TOOL_ARCH_H
guards, so they refer to the new architecture, rather than x86.

Comment all their contents out.

5 Try compiling. When it falls over on missing functions/types/constants, just
uncomment and fix up the copied ones. Just use stubs that fail (immediately
and obviously! — use the “I_die_here” macro) for functions and macros to get
things compiling.

For the kernel types, you’ll have to copy the types from the kernel source.
Use a recent kernel source, please. Don’t just pull in all the
corresponding types/macros that x86 provides, otherwise you might end up
providing more types/macros than the core actually needs; only pull in types
as the compiler asks for them. You’ll need a lot of the types in
vki_arch_posixtypes.h early on.

You’ll need to update the Makefile.am files if you add/remove files.

Once it compiles ok, get it to run:

6 Try running. When it falls over on stub function/macros, implement them
properly. The syscall table and syscall wrappers will be painful; do them
individually, on demand.

7 A lot of the arch-specific stuff has been abstracted out of the core, but
there undoubtedly remains some arch-specific stuff in there. Abstract it out
as necessary, updating the other archs appropriately.

8 If it crashes without telling you why, use lots of diagnostic printfs (or
OINKs) to track down the exact location of the crash.

Once it runs ok:

9 Add the arch to the tests/arch_test.c file so the reg test script will work.
(Don’t forget to add it to all_archs[].) Likewise for os_test.in and
platform_test.

10 Ensure the regression tests work, and add some arch-specific tests to
none/tests directory.

11 Add the relevant entries to valgrind.spec.in (copy the x86 and x86-linux
ones).

—————————————————————————–
Comments:
—————————————————————————–
1 As explained that these are mostly tests, they should be written for the platform, copying the existing linux ones and modifying them would be the way to start on them.

2 Extension of 1.

3 No need, because the architecture we’re porting to already has support.

4 Explained before in previous section’s points 3 and 4.

5 The compiling phase is almost over now. Instead of I_die_here macro, the vg_assert() macro was used to get things compiling.

The kernel types were implemented, although this version does indulge in pulling in the headers that provide types/macros which is more than the core needs. Specifically the vki-gnu.h file could be implemented in a way where the types are copied from the kernel headers and implemented in the header.

6-11 To be done. See TODO.

—————————————————————————–
TODO:
—————————————————————————–
Below I’ve listed all the nitty-gritty stuff that need to be implemented as
this project transitions from GSOC to GNU where I’m a contributor.

The compiling is about to be complete. The next steps would be:
1 The implementation of stuff where there are vg_assert()s. This would include only functions that are a part of the trap table. The types/constants are already dealt with in vki-gnu.h

The functions are another story. They belong to 3 broad categories – the traps and the functions originating from glibc. The traps are the ones that need to be wrapped so they are the final task. The glibc ones are what used to be wrapped in Linux as they were the syscalls there. Here in GNU Hurd, we have them available as part of glibc. So they can be called via interfaces generated by MIG.

2 Apart from these functions there are some functions that
a) implement a layer of abstraction to execute the traps and syscalls.
b) implement the wrapper framework
So they are another category that need attention.

3 There are many places where significant amount of assembly code occur. They are used in many of the aforementioned functions as inlines. They also occur in the wrapper machinery as assembly .S files.

4 The wrapping of traps.

—————————————————————————–
Comments:
—————————————————————————–
Funny thing is, my abstract focused on the syscall wrappers but it turned out that there were more aspects of Valgrind that needed attention first. I got to know about them firsthand through this GSOC Project.

Valgrind is an extensive and magnificent piece of software. Valgrind for Darwin took a really long time [1]. This just shows that a lot of dedication and effort is required to accomplish this. As, of now I believe I’ve done a fourth of the port.

Now, I’ve just started my third year of my Bachelors in Computer Science. I fully intend to see this project through, preferably before my graduation. Also, next year I intend to apply again for GSOC for a different project in either GNU Hurd, FreeBSD or any other microkernel organization.

As I was looking for more projects to work on, I was interested in the following projects – viengoos kernel, 64-bit version of GNU Hurd.

I may be incapable for those projects now, but I’ve got to learn stuff and start contributing small patches and someday be a maintainer for a rocking software project!

1 – http://blog.mozilla.org/nnethercote/2009/05/29/valgrind-on-windows/