This site has been retired. For up to date information, see handbook.gnome.org or gitlab.gnome.org.


[Home] [TitleIndex] [WordIndex] 

Remote execution testing

We intend to produce a test for BuildStream's remote execution system using docker images where possible. This will consist of:

This is a BuildStream-centric test, which means BuildStream is the thing being tested. Versions of BuildGrid and buildbox and bst-artifact-server will be fixed, while our CI will test the latest version of BuildStream against them. This may still reveal faults in either BuildGrid or buildbox, but I want to avoid updating the versions of all components at arbitrary intervals, because this makes comparing a test against previous results more difficult.

In the future, we may try to get other projects (e.g. BuildGrid) to run similar tests, in which case the current master of BuildGrid would be tested against fixed versions of BuildStream and BuildBox, hopefully reusing as much code as possible from our test suite.

Minimum viable test

The final piece - starting a bot - is slightly more tricky. We need to assemble buildbox ourselves and place it into a container which has BuildGrid in.

This can, I expect, be done by starting a second BuildGrid container with the entrypoint overwritten and using this container to download and build buildbox, then using this container as the build bot. If that container can be duplicated after building buildbox, and used to run multiple bots, all the better.

There is a Dockerfile present in buildbox, which means there have at least been attempts to make a buildbox docker image in the past. However, according to Jürg, this will only provide buildbox itself, not the BuildGrid buildbox bot.

What currently works

Configuring !BuildGrid and buildbox

These tests were run on the current master version of the BuildGrid repository (revision 33158b7f2baf28a89b5af6526fbadbebaf15e5ac).

Apply the following diff to DockerFile in the BuildGrid repository to build buildbox along with BuildGrid.

Note: these instructions are for an older version of buildbox, which used meson instead of cmake 

diff --git a/Dockerfile b/Dockerfile
index d533f68..5373604 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,8 @@
-FROM python:3.5-stretch
+FROM debian:buster
+
+RUN apt-get update && apt-get install -y \
+    python3 \
+    python3-pip
 
 # Point the path to where buildgrid gets installed
 ENV PATH=$PATH:/root/.local/bin/
@@ -13,10 +17,32 @@ WORKDIR /app
 COPY . .
 
 # Install BuildGrid
-RUN pip install --user --editable .
+RUN pip3 install --user --editable .
+
+WORKDIR /app
+
+RUN apt-get update && apt-get install -y \
+    gcc \
+    g++ \
+    git  \
+    grpc++ \
+    libfuse3-dev \
+    libssl-dev \
+    meson \
+    pkg-config \
+    uuid-dev \
+    && apt-get clean
+
+RUN git clone https://gitlab.com/BuildGrid/buildbox/buildbox-fuse.git /buildbox
+
+WORKDIR /buildbox
+
+RUN mkdir build && cd build && meson .. && ninja && ninja install && buildbox --help
+
+WORKDIR /app
 
 # Entry Point of the image (should get an additional argument from CMD, the path to the config file)
 ENTRYPOINT ["bgd", "server", "start", "-vv"]
 
 # Default config file (used if no CMD specified when running)
-CMD ["buildgrid/_app/settings/default.yml"]
+CMD ["tls.yml"]

Now add tls.yml, the alternative configuration file which configures TLS: 

server:
  - !channel
    port: 50051
    insecure-mode: false
    credentials:
      tls-server-key: /certs/server.key
      tls-server-cert: /certs/server.crt
      tls-client-certs: /certs/client.crt

description: |
  A single default instance.

instances:
  - name: ''
    description: |
      The main server

    storages:
      - !disk-storage &main-storage
        path: !expand-path $HOME/cas

    services:
      - !action-cache &main-action
        storage: *main-storage
        max-cached-refs: 256
        allow-updates: true

      - !execution
        storage: *main-storage
        action-cache: *main-action

      - !cas
        storage: *main-storage

      - !bytestream
        storage: *main-storage

Now build this with docker build --tag buildgrid_server ..

Now we can start our instances:

Create the openssl certificates somewhere: 

mkdir certs; cd certs
openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -batch -subj "/CN=bgd_server" -out "server.crt" -keyout "server.key"
openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -batch -subj "/CN=bgd_client" -out "client.crt" -keyout "client.key"
cd ..

Start the network:

docker network create buildgrid-net

Start BuildGrid server, using the 'certs' directory we created earlier: 

docker run -i -p 50051:50051 --network buildgrid-net --volume /home/jimmacarthur/bs/buildgrid/certs:/certs --name bgd_server buildgrid_server

Start a buildbox bot using the same image, but with a different entry point: 

docker run --entrypoint "bgd" --network buildgrid-net --volume /home/jimmacarthur/bs/buildgrid/certs:/certs buildgrid_server bot --remote https://bgd_server --client-key /certs/client.key --client-cert /certs/client.crt --server-cert /certs/server.crt --remote-cas https://bgd_server:50051 --cas-client-key /certs/client.key --cas-client-cert /certs/client.crt --cas-server-cert /certs/server.crt buildbox

This should produce a working remote execution service, accessible outside docker as https://localhost:50051. You'll need the certificates created earlier for your client configuration.

What needs doing

Things that would be nice to have

References:

[1] https://gitlab.com/BuildStream/buildstream-docker-images/issues/32

2024-10-23 11:36