Jonathan Banashek


Develop F# live in a Docker container

I think F# and Docker are both pretty cool technologies. Here is a way to create an awesome development environment with them.

Quick disclaimer:
Here are the different versions of the software that I'm using:

The finished code can be found https://github.com/Banashek/suavedockerdev

Project Scaffolding

To begin let's create a new directory, initialize our git repository, and create the new forge project.

projects % mkdir suavedockerdev
projects % cd suavedockerdev
suavedockerdev % git init
suavedockerdev HEAD % forge
Available commands:

  new [-n]: Create new project
  file [-f]: Adds or removes file from current folder and project.
  reference [-ref]: Adds or removes reference from current project.
  update: Updates Paket or FAKE
  paket: Runs Paket
  fake: Runs FAKE
  refresh: Refreshes the template cache
  help: Displays help
  exit [quit|-q]: Exits interactive mode

  --help [-h|/h|/help|/?]: display this list of options.

> new
Enter project name:
> sddweb
Enter project directory (relative to working directory):
>
Choose a template:
 - aspwebapi2
 - classlib
 - console
 - fslabbasic
 - fslabjournal
 - pcl259
 - servicefabrichost
 - servicefabricsuavestateless
 - sln
 - suave
 - suaveazurebootstrapper
 - websharperserverclient
 - websharperspa
 - websharpersuave
 - windows

> console
Generating project...

Although there is a suave template, I chose to use the console library so that I could understand the basic pieces of F# and Suave before moving on to a pre-built template.

Before you commit the generated project scaffolding, you can choose whether you want to add the packages folder to your gitignore file or not. Packaging compiled dependencies versus having declarative dependency files is a debate that is out of the scope of this post. I am going to choose to add it to the gitignore file so that the program will download the needed dependencies when the container is being built.

suavedockerdev HEAD % echo "packages/" > .gitignore
suavedockerdev HEAD % git add .
suavedockerdev HEAD % git commit -m "Initial commit of generated project"

If we want to run this new project we can simply build and then run the compiled executable located in the `build/` folder.

suavedockerdev master % sh build.sh
suavedockerdev master % mono build/sddweb.exe test
[|"test"|]

This is going to be a Suave app, so lets add in the dependency using forge (forge will automatically add the reference to the .fsproj file so we don't have to). We will also add in the FSharp.Compiler.Service dependency so that we can send updated files to be compiled and reload the application.

suavedockerdev master % forge paket add nuget Suave project sddweb
suavedockerdev master % forge paket add nuget FSharp.Compiler.Service project sddweb

Now that we have our dependencies installed, we can move on to the core files that will enable us to do interactive development: app.fsx, build.fsx, the Dockerfile, and the docker-compose.yml file.

The app.fsx file will live in the ssdweb/ folder and will be the entrypoint to our Suave application.
The build.fsx will be our FAKE build script responsible for watching the project directory for changes and restarting the application (app.fsx) when a file is changed.
The Dockerfile represents the steps required to build our application in a single docker container.
The docker-compose.yml file represents the file we will use to start up our application in docker-compose. Running our app via docker-compose will allow us to easily add containers for other useful services which our application can utilize to provide more functionality (think databases, cache servers, etc).

app.fsx

#r "../packages/Suave/lib/net40/Suave.dll"
open Suave
open Suave.Successful

let app = OK "Hello Suave"

app.fsx lives in the fsproj directory (`suavedockerdev/sddweb/app.fsx`) is just the hello world example from their website for now.

build.fsx

#r "./packages/FSharp.Compiler.Service/lib/net45/FSharp.Compiler.Service.dll"
#r "./packages/Suave/lib/net40/Suave.dll"
#r "./packages/FAKE/tools/FakeLib.dll"
open Fake
open System
open System.Net
open System.IO
open System.Threading
open Suave
open Suave.Web
open Microsoft.FSharp.Compiler.Interactive.Shell

let sbOut = new Text.StringBuilder()
let sbErr = new Text.StringBuilder()

let fsiSession =
  let inStream = new StringReader("")
  let outStream = new StringWriter(sbOut)
  let errStream = new StringWriter(sbErr)
  let fsiConfig = FsiEvaluationSession.GetDefaultConfiguration()
  let argv = Array.append [|"/fake/fsi.exe"; "--quiet"; "--noninteractive"; "-d:DO_NOT_START_SERVER"|] [||]
  FsiEvaluationSession.Create(fsiConfig, argv, inStream, outStream, errStream)

let reportFsiError (e:exn) =
  traceError "Reloading app.fsx script failed."
  traceError (sprintf "Message: %s\nError: %s" e.Message (sbErr.ToString().Trim()))
  sbErr.Clear() |> ignore

let reloadScript () =
  try
    traceImportant "Reloading app.fsx script..."
    let appFsx = __SOURCE_DIRECTORY__ @@ "/sddweb/app.fsx"
    fsiSession.EvalInteraction(sprintf "#load @\"%s\"" appFsx)
    fsiSession.EvalInteraction("open App")
    match fsiSession.EvalExpression("app") with
    | Some app -> Some(app.ReflectionValue :?> WebPart)
    | None -> failwith "Couldn't get 'app' value"
  with e -> reportFsiError e; None

let currentApp = ref (fun _ -> async { return None })

let serverConfig =
  { defaultConfig with
      homeFolder = Some __SOURCE_DIRECTORY__
      logger = Logging.Loggers.saneDefaultsFor Logging.LogLevel.Debug
      bindings = [ HttpBinding.mk HTTP (IPAddress.Parse "0.0.0.0") 8083us] }

let reloadAppServer (changedFiles: string seq) =
  traceImportant <| sprintf "Changes in %s" (String.Join(",",changedFiles))
  reloadScript() |> Option.iter (fun app ->
    currentApp.Value <- app
    traceImportant "Refreshed app." )

Target "run" (fun _ ->
  let app ctx = currentApp.Value ctx
  let _, server = startWebServerAsync serverConfig app

  // Start Suave to host it on localhost
  reloadAppServer ["app.fsx"]
  Async.Start(server)

  // Watch for changes & reload when app.fsx changes
  let sources = 
    { BaseDirectory = __SOURCE_DIRECTORY__
      Includes = [ "**/*.fsx"; "**/*.fs" ; "**/*.fsproj"; ]; 
      Excludes = [] }

  use watcher = sources |> WatchChanges (Seq.map (fun x -> x.FullPath) >> reloadAppServer)
  traceImportant "Waiting for app.fsx edits. Press any key to stop."
  // Hold thread open so that docker doesn't close the process when it detaches in compose
  Thread.Sleep(-1)
)

RunTargetOrDefault "run"

build.fsx lives in the top level directory (`suavedockerdev/build.fsx`) and is the core of how the hot-reloading of our application works.

While other languages have their own programs for hot-reloading changed files (nodemon for node, dotnet-watch for .net core, etc), this is something that is written by hand in our build script.

This file is mostly taken from Tomas Petricek's found here: https://github.com/tpetricek/suave-xplat-gettingstarted/blob/master/build.fsx Not only did he create the base for this file, but also helped me figure out how to get it working in docker-compose!

This file works by doing a few things:

First it sets up a F# compiler service session named `fsiSession`. This allows us to send changed code to be compiled on a different thread and returned to us in it's final form.

It then creates the `reportFsiError` helper function for writing errors to output should there be an issue.

Next it defines the `reloadScript` function. This function defines `appFsx` to represent the app.fsx file. If your file is in a different location, make sure this is pointing to the correct spot! It uses the `EvalInteraction` and `EvalExpression` to re-evaluate the app.fsx file. It's basically a hand made repl for your app.fsx file.

`currentApp` is defined as a mutable (ref) placeholder for your app so that once the compiler service evaluates your new interpereted app.fsx file, that it can re-assign the value of `currentApp` to the result of the evaluation.

`serverConfig` is then defined with a few differentiations from the Suave default config. One special thing to note here is that I used the ip `0.0.0.0`. Docker by default binds all available interfaces (INADDR_ANY) at the address 0.0.0.0 unless configured to do otherwise. I wanted to leave that alone, so I just made the change in the build file.

The `reloadAppServer` function comes next. It takes the sequence of changed files, logs a message, calls the `reloadScript` function, then swaps in the new app value.

The last piece of this script is the actual build target. This is what is called when FAKE is executed by the `build.sh` script. First it calls an initial `reloadAppServer` with the app.fsx file. It then starts the server asynchronously with the initial value returned. Next it sets the source directories variable to include .fs, .fsx, and .fsproj files to be watched. Once the directories are set, it creates a `System.IO.FileSystemWatcher` and calls the `WatchChanges` method to watch those files for changes, composing the `reloadAppServer` function on to the end to be called whenever a file changes. It then calls `Thread.Sleep(-1)` to hold the thread open so that when the application is run in a detatched container (docker-compose) that it continues to keep the process open.

Local Live Editing

With these two files in place, you should now be able to live edit your web server code.

To do this open up two terminals. In the first terminal run `sh build.sh` from the projects root directory. The output should look something like this:

suavedockerdev master % sh build.sh
No version specified. Downloading latest stable.
Paket.exe 2.66.8.0 is up to date.
Paket version 2.66.8.0
0 seconds - ready.
Building project with version: LocalBuild
Shortened DependencyGraph for Target run:
<== run

The resulting target order is:
 - run
Starting Target: run
Changes in app.fsx
Reloading app.fsx script...
Refreshed app.
dirs to watch: ["/Users/jonbanashek/Projects/fs/suavedockerdev";
 "/Users/jonbanashek/Projects/fs/suavedockerdev";
 "/Users/jonbanashek/Projects/fs/suavedockerdev"]
watching dir: /Users/jonbanashek/Projects/fs/suavedockerdev
[I] 2016-06-02T23:47:35.6877090Z: listener started in 972.925 ms with binding 0.0.0.0:8083 [Suave.Tcp.tcpIpServer]
watching dir: /Users/jonbanashek/Projects/fs/suavedockerdev
watching dir: /Users/jonbanashek/Projects/fs/suavedockerdev
Waiting for app.fsx edits. Press any key to stop.

Now in your second terminal, you should be able to run curl to see the output.

suavedockerdev master % curl localhost:8083
Hello Suave

Now in whatever editor you're using, go and change the string in the `app.fsx` file to "Hello Suave Interactive!"

If you go back to the first terminal, you should see output indicating that the watcher process noticed the file change and reloaded the application.

Changes in /Users/jonbanashek/Projects/fs/suavedockerdev/sddweb/app.fsx
Reloading app.fsx script...
Refreshed app.

And if we run curl again we should see the new output!

suavedockerdev master % curl localhost:8083
Hello Suave Interactive!

Setting up Docker for in-container development

Live-reloading of an app is cool, but that could break if I tried it on a friend's machine who has an old copy of xamarin, unity3d, brew, and 3 other versions of mono linked in weird ways which they forgot about and have never gone back to check on since they never use it themselves and all the apps that need it are working at the moment. It would be pretty sweet if I could tell them that all they needed to do to start playing around with the project was `git clone` and a `docker run` command. I would also sleep easy knowing that any other places I would run it (CI server, Azure, AWS, DigitalOcean, etc) would have the same output. So let's get this working in docker.

All that we need to do is create a Dockerfile and pass some parameters to the `docker run` command to get this to work. To be fair, we did do a little bit of setup earlier with the specific IP address and Thread.Sleep call that were in the `build.fsx` file.

The Dockerfile

FROM fsharp/fsharp:latest
COPY . /app/
WORKDIR /app
EXPOSE 8083
CMD ["/bin/bash", "build.sh"]

The Dockerfile is pretty simple.

It uses the fsharp image as it's base (which takes care of making sure mono and other fsharp dependencies are present)

Then it copies the application code over to the `/app/` directory of the container (absolute path).

The `WORKDIR` command changes the directory into the copied application directory.

We `EXPOSE` the 8083 port from the container to the container host so that there is a way to communicate with the application.

Finally we run our same build script as the entrypoint into our docker container.

Docker Interactive

Now that the docker file is set up we can do the same 2 terminal setup from earlier.

First let's build the container into a container image that docker can run. We'll also give it a tag with a specific version so that we can keep track of our image.

suavedockerdev master % docker build -t sdd:0.1 .
Sending build context to Docker daemon 166.4 MB
Step 1 : FROM fsharp/fsharp:latest
 ---> 73ffae16882d
Step 2 : COPY . /app/
 ---> 54e8bfacda51
Removing intermediate container e5aee6cb48ad
Step 3 : WORKDIR /app
 ---> Running in 79d24650eb26
 ---> f38224a4e00c
Removing intermediate container 79d24650eb26
Step 4 : EXPOSE 8083
 ---> Running in c3eebcac1cdb
 ---> 13ab455b1004
Removing intermediate container c3eebcac1cdb
Step 5 : CMD /bin/bash build.sh
 ---> Running in a16513b815b7
 ---> 198818469fe1
Removing intermediate container a16513b815b7
Successfully built 198818469fe1

Now that we have a container image built we can run the docker image in terminal 1.

suavedockerdev master % docker run -it --rm -v `pwd`:/app -p 8083:8083 sdd:0.1
No version specified. Downloading latest stable.
Paket.exe 2.66.8.0 is up to date.
Paket version 2.66.8.0
Downloading FAKE 4.28
Downloading FSharp.Compiler.Service 3.0
Downloading FSharp.Core 4.0.0.1
Downloading Suave 1.1.2
12 seconds - ready.
Building project with version: LocalBuild
Shortened DependencyGraph for Target run:
<== run

The resulting target order is:
 - run
Starting Target: run
Changes in app.fsx
Reloading app.fsx script...
Refreshed app.
dirs to watch: ["/app"; "/app"; "/app"]
watching dir: /app
[I] 2016-06-03T01:45:10.3901380Z: listener started in 1674.804 ms with binding 0.0.0.0:8083 [Suave.Tcp.tcpIpServer]
watching dir: /app
watching dir: /app
Waiting for app.fsx edits. Press any key to stop.

This docker command has a few parameters which are important:

If you go do the curl->edit->curl test, you should see that you can now interactively develop inside of a reusable docker container!

Live Editing in Docker Compose

Docker itself gives us a utility for reliable code execution. Docker compose takes that up a level to wire multiple docker containers together to allow you to bring up a reliable infrastucture. First we'll create the docker-compose.yml file that will allow us to live-edit in docker compose, and then we'll see how docker compose allows us to easily add other utilities to our applications infrastructure.

docker-compose.yml

web:
  build: .
  working_dir: /app
  ports:
    - "8083:8083"
  volumes:
    - .:/app

This file declares a volume `web` which is built from the current directory, and then passes the same parameters that we were passing when running `docker run`.

Run `docker-compose up` to pull up our container.

suavedockerdev master % docker-compose up
Building web
Step 1 : FROM fsharp/fsharp:latest
 ---> 73ffae16882d
Step 2 : COPY . /app/
 ---> a93621673380
Removing intermediate container 4d74d0de4a28
Step 3 : WORKDIR /app
 ---> Running in 4114058dd792
 ---> ef52c5f3b7d8
Removing intermediate container 4114058dd792
Step 4 : EXPOSE 8083
 ---> Running in 91a922a5aff6
 ---> ce5eefc7f3f9
Removing intermediate container 91a922a5aff6
Step 5 : CMD /bin/bash build.sh
 ---> Running in d417522b6b05
 ---> 17f760a72dfb
Removing intermediate container d417522b6b05
Successfully built 17f760a72dfb
WARNING: Image for service web was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating suavedockerdev_web_1
Attaching to suavedockerdev_web_1
web_1  | No version specified. Downloading latest stable.
web_1  | Paket.exe 2.66.8.0 is up to date.
web_1  | Paket version 2.66.8.0
web_1  | Downloading FAKE 4.28
web_1  | Downloading FSharp.Compiler.Service 3.0
web_1  | Downloading FSharp.Core 4.0.0.1
web_1  | Downloading Suave 1.1.2
web_1  | 14 seconds - ready.
web_1  | Building project with version: LocalBuild
web_1  | Shortened DependencyGraph for Target run:
web_1  | <== run
web_1  |
web_1  | The resulting target order is:
web_1  |  - run
web_1  | Starting Target: run
web_1  | Changes in app.fsx
web_1  | Reloading app.fsx script...
web_1  | Refreshed app.
web_1  | dirs to watch: ["/app"; "/app"; "/app"]
web_1  | watching dir: /app
web_1  | [I] 2016-06-03T02:41:09.2202990Z: listener started in 1425.439 ms with binding 0.0.0.0:8083 [Suave.Tcp.tcpIpServer]
web_1  | watching dir: /app
web_1  | watching dir: /app
web_1  | Waiting for app.fsx edits. Press any key to stop.

Once more curl->edit->curl and see that we're good to go.

Reaping the rewards

We could have settled with the local live-editing for developing our Suave application, but we decided to get it set up in docker compose with some extra work.

Let's see how easy it is to add a redis box to our development environment.

docker-compose.yml

web:
  build: .
  links:
    - redis:redis
  working_dir: /app
  ports:
    - "8083:8083"
  volumes:
    - .:/app

redis:
  restart: always
  image: redis:latest
  ports:
    - "6379:6379"
  volumes:
    - redisdata:/data

That's it, we can just run `docker-compose up` again and now our Suave app has a redis app which it can communicate with.

Now eventually we will want to have another build target so that we have something that can run in production, but this works for fleshing out ideas for now.

Thanks to Tomas Petricek and Reed Copsey, Jr. for the help they provided over on the #fsharp-beginners channel of the functionalprogramming slack!

linkedin-square github-square