Thomas Letan's Blog - ocaml

Peer-Programming in Modern OCaml with ChatGPT and Gemini

June 2, 2025

Peer-Programming in Modern OCaml with ChatGPT and Gemini

It is June 2025, and LLMs are everywhere and do everything now. I have never been a diligent adopter of them myself. The past few months, I started to feel a bit “left out,” though. Colleagues and friends are starting to integrate LLM-powered tools into their personal toolkit, with notable successes.

Early May, I decided to challenge myself to implement a simple tool to generate a summary from YouTube videos using Vosk for speech recognition and Ollama for generating summaries using LLMs running locally. I could hit two birds with one stone—experimenting with LLMs to write and power software.

I decided to implement as much as possible in OCaml, for two main reasons. Firstly, this is the main language I use at $WORK. I wanted to get a sense of how LLMs could help with the software stack I used 7+ hours a day. Secondly, it was a good opportunity to catch-up with the OCaml 5 ecosystem (Eio in particular).

This write-up is a sort of dev log of this exercise. Its main focus is not to explain in depth the code I ended up writing, but rather to recollect on my wins and losses in adding LLMs in my developer toolkit.

TL;DR

In this article, I am using “Tip” blocks to highlight my key findings and lessons learned. That being said, for readers in a hurry, here’s how ChatGPT summarizes these blocks.

Prompting is a skill that improves through trial and error—many failed prompts help build intuition.
LLMs may suggest non-existent functions; using LSP tools helps identify these quickly.
Standard formats like WAV lead to more accurate LLM outputs.
LLMs without session memory tend to repeat mistakes; shared context is important.
Structuring commit message prompts (e.g., What / Why / How) produces consistently good results.
LLMs struggle with libraries like Eio, possibly due to name ambiguity or unstable APIs.
Providing project-specific context (e.g., via direnv) is likely to help reduce repeated hallucinations.
Prompting LLMs for MR descriptions or commits can eliminate empty submissions and speed up review.

You should still definitely read the full piece, though. I don’t think my prompt was particularly good 🤫.

Editor Integration

My first task was to grant myself the ability to leverage LLMs from my editor. I had been using the web chat of ChatGPT for a while, but it now felt antiquated since I had seen a freshly hired coworker get ChatGPT to generate for themselves a dozen tests directly from VS Code.

I have returned to Neovim for a few years, and I am not ready to migrate to VS Code. I would have been surprised if the Vim/Neovim communities wouldn’t have a viable plugin for me, though.

I asked both ChatGPT and Gemini to find my candidates, but the plugins they suggested seemed unmaintained, often outdated.

In the end, I found CodeCompanion.nvim by myself, through a good old Google research. I asked ChatGPT why it hadn’t suggested it to me, and it seems like my prompt were biased. By asking for “a Neovim ChatGPT plugin” or “a plugin to integrate Gemini to Neovim,” I had unnecessarily narrowed the LLM scope.

Tip

I guess one does not become a prompt engineer in a day. This is actually one of the reasons I want to use LLMs more seriously. To build myself intuitions of which prompts work and which don’t. After this project, I have mostly uncovered a bunch of the latter category 😅.

@yurug had told me he was impressed by Gemini Pro, so I decided to make it the default adapter for the CodeCompanionChat command. I tried to make Gemini Pro the default model for this adapter, it was challenging and LLMs weren’t able to help. When I finally found the correct setup option, it turns out I hadn’t generated a token allowing me to use Pro.

Well. That gave me the opportunity to benchmark Gemini Flash, then.

Speech Recognition with Vosk

ChatGPT suggested Vosk as a way to get a transcript of an audio file, so it was also a good opportunity to write bindings (something I had dodged for a long time for no particular reason).

As of June 2025, there is no OCaml bindings for the Vosk API , so my first task was to write my own as part of a project soberly called ocaml-vosk .

Gemini Flash was able to help me understand how ctypes and ctypes.foreign works. This was my first experience interacting with an LLM from my Neovim window, and it was pretty convincing. It gave me the opportunity to learn that one can declare opaque types in OCaml (not just via mli files). It makes sense, but it was news to me.

Then, Gemini suggested me to use EIO’s Switch to deal with automatic memory management (in place of Gc.finalise). It was the first time I heard about it, and the fact that I learned their existence from the perspective of resource management (not fiber management) was a good accident.

The first point of friction came when I started build a high-level interface for my Vosk bindings. More specifically, given a Cstruct.t value, how do I get a pointer and a length? It turns out that while both ChatGPT and Gemini Pro know how to do so, Gemini Flash hallucinates every step of the way.

The solution is actually pretty straightforward.

let ptr =
  Ctypes.bigarray_start
    Ctypes.array1
    (Cstruct.to_bigarray buffer)
in
let len = buffer.Cstruct.len in

Gemini Flash kept suggesting I use Ctypes.ptr_add instead, though. Don’t search for it, it does not existWhile reviewing this article, ChatGPT gently hinted that while ptr_add does not exist, Ctypes.(+@) does. . When I suggested Cstruct.to_bigarray, it warned me about the fact that this call would create a copy of the underlying buffer. ChatGPT and Gemini Pro disagreed, and I could convince myself that they were right by looking at the code. Interestingly, I was also able to convince Gemini Flash it was wrong by copy/pasting the relevant code snippet.

Tip

Having an LLM suggesting you to use a function which does not exist is very frustrating. Especially if it happens several times in a row—it recognizes its mistake and proposes an alternative that is as nonexistant as the first one. At least, with LSP it is pretty straightforward to know when it happens.

Using Vosk is one thing, but then I couldn’t find any OCaml package to read audio files compatible with Vosk expectations. Implementing what I needed in OCaml gave me more opportunities to learn about EIO, but most importantly, it showed how having a chat with an LLM directly from my editor was convenient. I was able to learn about WAV files, RIFF header and subchunks and PCB 16-bit mono audio data without leaving Neovim. And by giving Gemini access to my buffer, I troubleshot most of my issues fairly quickly (except when they were EIO-specific—more on that later).

Tip

For widespread encoding like WAV files, LLMs shine particularly bright.

In the end, EIO-specific code put aside, this task was roughly solved by (1) writing bindings for the few functions of the Vosk API I needed, and (2) translating C examples provided by Gemini into good-looking OCamlIt’s a little out of scope for this article, but I discovered when writing the high-level API for Vosk that Switches are very easy to misuse. It is as simple as (incorrectly) turning an eager function consuming a buffer into a Seq-based alternative, while forgetting the use of Switch.run on top of the function. .

Witnessing my example program outputting the transcript of audio files as it was processing them felt pretty good, and I was soon ready to tackle the second part of this project: prompting a LLM to summarize it.

Prompting Local LLMs with Ollama

Similarly to Vosk, there is no on the shelf package available to use Ollama from an OCaml program. As a consequence, I created a second repository (ocaml-ollama if you can believe it).

How It Started

Turns out, you don’t use Ollama the same way you use Vosk. The latter is a C library that you can call from your binary, the former actually uses a client/server architecture. I asked LLMs what was the best solution for performing HTTP requests with Eio, and cohttp-eio came back as a good candidate. I’m already familiar with cohttp, since we are using it at $WORK, but it’s actually a transitive dependency (of a framework called resto ).

I am actually a little frustrated with resto, so I welcomed the opportunity to familiar myself a little more with cohttp directly. I quickly implemented the helper fetching the list of models available from a given Ollama instance.

Then, I got myself side tracked.

More LLMs Lies

Persistent HTTP connections are a pet peeve of mine. Establishing a TCP connection, negotiating TLS encryption, all of that takes time—creating a new socket for each request a daemon really frustrates me as a result.

So I asked.

Does cohttp-eio reuses already established connections when performing two requests on the same host?

ChatGPT 4o. Gemini 2.5 Flash. Gemini 2.5 Pro. They all assured me it was the case, as long as I was careful and reused the same Cohttp_eio.Client.t instance. For instance, here is the first few words of ChatGPT when prompted with this question.

As of current behavior in cohttp-eio-client, yes, it does reuse already established connections when making multiple requests to the same host, provided certain conditions are met.

It’s a lie. Don’t trust them. They don’t reuse existing HTTP connection.

I was very doubtful, so I asked them how to check this. tcpdump was mentionedI later discovered eio-trace and it would have been much more straightforward to use this tool to inspect Cohttp_eio.Client’s default behavior. No LLM thought of that, sadly. . I got traces I couldn’t read at first glance, so I just copy/pasted them to the LLMs… and sure enough, they confirmed what I suspected. Cohttp_eio.Client does not share connections by default. It creates a socket for each request.

It’s actually pretty easy to convince yourself that it is the case by reading the implementation of Cohttp_eio.Client .

type connection = Eio.Flow.two_way_ty r
type t = sw:Switch.t -> Uri.t -> connection

(* simplified version of [make], omitting the support for HTTPS *)
let make () net : t = fun ~sw uri ->
  (Eio.Net.connect ~sw net (unix_address uri) :> connection)

There is nothing here dealing with persistent connections. Eio.Net.connect uses a switch for resource management, but does not perform any kind of connection caching.

That’s okay, though. Yak shaving is a real thing. I can stop working on my Ollama client library for a while, just to fix this.

The Questionable Side Quest of Implementing a Connection Pool for `cohttp-eio`

The bottom-line of this little adventure is: I should have updated my default prompt to remind the LLMs that Cohttp_eio.Body.drain in not a thing.

But let’s start from the beginning. Over the course of a few days, I have successfully implemented a wrapper on top of Cohttp_eio.Client to deal with persistent connections. It’s not rocket science, but it’s still a subtle endeavor, which necessitated a good understanding of Eio and cohttp. I cannot say LLMs were instrumental for the task. They gave me good pointers to start from, but they also misled me a bunch of times.

Sometimes, the help came in surprising ways. One anecdote in particular stuck with me. I decided I needed a get operation for Eio.Pool pools, which sadly only proposes use.

(* Provided by Eio.Pool *)
val use : 'a t -> ('a -> 'b) -> 'b

(* Not provided *)
val get : sw:Switch.t -> 'a t -> 'a

The key insight is that get allows callers to pick something from the pool, and only put it back when the switch is released.

My first implementation of get was roughly as followsI didn’t even consider asking an LLM to propose me an implementation, now that I think about it. I really am no vibe coder yet. .

open Eio.Std

let get ~sw t =
  let x, rx = Promise.create () in
  let never, _ = Promise.create () in
  Fiber.fork ~sw (fun () ->
      Eio.Pool.use t @@ fun conn ->
      Promise.resolve rx conn;
      Promise.await never);
  Promise.await x

And it didn’t work. The resulting program was hanging, because of how Fiber.fork ~sw works. Basically, the fiber created by fork becomes part of the set of fibers the switch sw waits for. Since, in my case, said fiber would never be resolved, I had created a deadlock.

I asked Gemini Pro 2.5 for help, and out of curiosity, I looked at its reasoning steps. Very early on, it mentioned Fiber.fork_daemon, but surprisingly Fiber.fork_daemon was not mentioned in the final answerOnce again, I had asked the wrong question. I asked for the Fiber equivalent of Lwt.async. I had overlooked that Lwt.async had a very particular behavior wrt. exceptions, that Gemini Pro tried very hard to replicate. I didn’t care at all about the exceptions I could raise, here! . Have I not been curious at that time, I would have missed the correct solution@alice provided me the answer a few minutes later, so I’d have been fine in the end 😅. .

I think my experience overall was made a little more frustrating than it should have been because I have never constructed a “context” that I could share between coding sessions. I haven’t enabled the memory saving setting in ChatGPT. Besides, everytime I opened Neovim, Gemini was starting from scratch. I should try to change that, to prevent the LLMs from doing the same mistakes again and again—typically, the Cohttp_eio.Body.drain function they kept bringing up.

Tip

I need to investigate how I can specialize my default prompt for each software project I am working on. I imagine I can rely on an environment variable and direnv .

Finally, it’s when I worked on this library that I came up with a nice prompt for Gemini to write my git commit messages for me.

@editor #buffer Add a git commit title and message. Structure the description in three sections (What, Why, How). Wrap the sections at 72 columns. Don’t forget the git title, and always insert a new line between the title and the description.

This prompt gives pretty cool result. It is still necessary to review it, because in a few instances I caught false statement in the proposal. But overall, it gives really meaningful output. Almost all commits of the library have been written with this prompt.

Tip

If anything, I don’t think I will never open a Merge Request with an empty description ever again.

And that, kids, is how I released cohttp-connpool-eio.0.1 .

Wrapping-up a Minimal Ollama Chat

Integrating cohttp-connpool-eio in my ocaml-ollama project led me to find a bug in the former. More specifically, the Cohttp_connpool_eio.warm function that can be used to pre-populate a new pool was doing so by performing a specified HEAD request to the host as many time as the pool sizeIn a later iteration of the library, warm only establishes connections, and does not perform any unnecessary HTTP requests. .

It worked well against both https://www.google.com and https://soap.coffee/~lthms, but when I tried with the Ollama server, it decided to hang. Why?

Well, I tried asking my new friends the LLMs, but didn’t get any answer I felt confident with. At this point, my trust in their EIO expertise was rather low, and I was more skimming through their answer to find a lead I would follow myself than anything else. In the end, I completely dropped the LLMs here, and went back to what I usually do: experimenting, and reading code.

I reproduced the issue with curl: curl -X HEAD hangs as well with Ollama, while curl --head does not. The former tries to read the response body, based on the response headers (e.g., content-length). The latter doesn’t, because it knows HEAD always omits the body. I am not sure why the hanging behavior does not show for curl -X HEAD https://www.google.com, though.

But anyway, once the bug was fixed, I could return to playing with Ollama.

I then decided to implement a helper to call POST /api/generate . It is the simplest way with Ollama to generate an LLM’s answer from a prompt. Interestingly enough, it is a “streamed” RPC using the application/x-ndjson content type. Instead of computing the answer before sending it to the client, the server instead sends JSON-encoded chunks (transfer-encoding: chunked ).

I tried to implement that with cohttp-eio, and it failed miserably with obscure parsing error messages.

After a bit of debugging, it became clear that Eio.Buf_read.parse was not behaving as I thought it was, which made me feel paranoid about how cohttp-connpool-eio handles connection releases. In the end, I had to unpack how the Cohttp_eio.Body.t work under the hood wrt. End_of_file to move on. Once again, my LLM friends weren’t particularly helpful: they were hallucinating Buf_read functions, and never considered to mention that parse only works for complete response.

Tip

My personal conclusion is that ChatGPT and Gemini quickly show their limits for non-trivial programming task involving Eio and its ecosystem. I am really curious to understand why. Do they keep hallucinating functions because Eio is a really generic name, and maybe they are mixing context from the Python library with the OCaml one? Or is it because the API of Eio has changed a lot over the years?

I am also wondering how, as a the author of a library, I can fix a similar situation. Assuming ChatGPT starts assuming false statements about cohttp-connpool-eio for instance, how do I address this? I suspect being “LLMs-friendly” will be increasingly important for a software library’s success.

In the end, ChatGPT and Gemini were just another source of inputs, not the main driver of my development process.

Putting Everything Together

Turns out, you really need just one RPC to generate a summary for a text input, so it wasn’t long before I could chain everything. I pulled mistral:7b-instruct-v0.2-q4_K_M (over a suggestion by ChatGPT, if I remember correctly), and got a summary from the video I had downloaded.

Just kidding. Out of nowhere, I decided to pursue yet another side quest, and gave a try to the fancy dune pkg lock command. Then I was able to generate my summary, using the following prompt.

Generate a summary of the raw transcript of a video provided after
this paragraph. The transcript may be in a language that is not
English, but the summary should always be in English. You should
adopt a neutral point of view (i.e., even if the transcript speaks
in the first person, you should always use the third person). Each
line is an utterance. Keep the summary short and engaging, your
goal is to provide a good overview of what was said.

----

{Vosk output}

And with this, it was time to wrap-up. And what better way to do so than to write this little journal entry? So I did, and when I was half way through my first draft, I fired a new chat buffer to ask for advice from my new friend Gemini Flash.

#buffer Here is a very preliminary, incomplete draft of a blogpost. Can you try to anticipate if it will find an audience?

It wasn’t long before Gemini turned me down.

My apologies, but I cannot anticipate whether this blog post will find an audience. My capabilities are focused on programming-related tasks like code explanation, review, generation, and tool execution within the Neovim environment.

Let me know if you have any questions about the code itself, or need assistance with Neovim.

🥲Fortunately, ChatGPT was less opinionated. .

Final Words

Although I had already used ChatGPT and other models in the past, this was the first time I tried to make them a central part of my workflow. I learned a lot during this experiment, and I now have an integrated setup I enjoy using.

I need to keep digging. Try more models (there are a lot of those now). And get better at writing good prompts which do not lead the LLMs astray. They are here to stay, after all. I better learn how to take the most from them.

Serving This Article from RAM for Fun and No Real Benefit

December 25, 2024

Serving This Article from RAM for Fun and No Real Benefit

docker meta ocaml

In 2022, Xe Iaso published a transcript of their talk on how their website was working at the time . In a nutshell, their approach consisted of a server preprocessing the website from its source at startup, then serving its contents from memory. If you have not already, I can only encourage you to read the article or watch the talk, as the story they tell is very interesting. For me personally, it sparked a question: what if, instead of preprocessing the website at startup, one decided to embed the already preprocessed website within the program of the HTTP server tasked to serve it?

Fast-forward today, and this question has finally been answered. The webpage you are currently reading has been served to you by an ad hoc HTTP server built with Dream , whose binary is the only file I need to push to my server to deploy the latest version of my website. I have actually deployed it, and it’s been serving the contents of this website for more than a week now.

What did I learn from this fun, little experiment? Basically, that this approach changes nothing, as far as Lighthouse and my monitoring is concerned. I couldn’t find any meaningful differences between a static website served by Nginx, a piece of software with thousands and thousands of engineering work behind it, and my little toy web server pieced together in an hour or so. Still. It was fun, so why not write about it?

This article is a kind of experience report. I’ll dive into what I have done to turn my website into a single, static binary. Not only does it mean writing some OCaml, which is always fun, but it also requires understanding a little some key HTTP headers, as well as using Docker to build easily deployable binaries. All in all, I hope it will be an interesting read for the curious minds.

Embedding My Website in a Binary

Not much had changed much since I stopped using cleopatra to generate this website, and the article I published in 2023 still stands. In a nutshell, I work in the site/ directory, and soupault generates my website in the out/~lthms directory, thanks to a collection of built-in and ad hoc pluginsFor instance, Markdown footnotes are turned into side notes with a soupault plugin. . To deploy the website, I was relying on rsync to sync the contents of the out/~lthms directory with the directory statically served by a Nginx instance on my personal server.

The first step of my little toy project is to actually embed the output of soupault into an OCaml program.

That’s where ocaml-crunch comes in handy. It is a MirageOS project, whose only job is to generate an OCaml module from a file system directory. It is straightforward to use it from Dune.

; file: out/dune
(rule
 (target website_content.ml)
 (deps (source_tree ~lthms))
 (action
  (run ocaml-crunch -m plain -o %{target} -s ~lthms)))

This snippet generates the website_content.ml module, which we can then expose through a library with the library stanza.

; file: out/dune
(library
 (name website_content))

And we are basically done. Excluding an Internal module, the signature of Website_content is pretty straightforward.

val file_list : string list
val read : string -> string option
val hash : string -> string option
val size : string -> int option

Serving the content with Dream

Dream is a cool project, and provides a straightforward API that we can leverage to turn our list of in-memory files into an HTTP server.

Naive Approach

Our goal now is to create a Dream.handler for each item in file_list. Done naively (as was my first attempt), it gives you something of the form:

let make_handler ~content path =
  Dream.get path (fun req ->
    Lwt.return (Dream.response content)))

Which we can use to build the main route we will then pass to Dream.router.

let website_route =
  Dream.scope "~lthms" []
  @@ List.map
       (fun path ->
         let content = Option.get (Website_content.read path) in
         make_handler ~content path)
       Website_content.file_list

With this approach, we build our handlers once, and then the lookup is done by Dream’s router. It could be an interesting experiment to see if doing the lookup ourselves is more performant (since Dream’s router is very generic, while in our case we don’t really need to parse anything). I remember Xe routing is basically going through a linked list, which seems strange at first, but works very well in practice because they have ordered said list with the most recent articles up front, and everybody comes to their website to read the latest article anyway.

It does not take an extensive QA process to figure out that this approach is far from being enough. To name a few things:

My website assumes http://path/index.html can be accessed with http://path/ or http://path. Our little snippet does not handle this.
Browsers expect the Content-Type headers to be correctly set. To give an example, they won't load a CSS file if the Content-Type header is not set to text/css.
Browsers work best for websites that take the time to provide caching directives. Our little snippet does not care to do so.
Even if my website is rather lightweight20MBytes at the time of writing the first version of this article. , compressing the response of our HTTP server for clients that support it is always a good idea.

This is a gentle reminder of all the things Nginx can do for you with very little configuration.

Handling `index.html` Synonyms

This one is rather simple. For files named index.html, we need 3 handers, not just one. We can achieve this with an additional helper make_handler_remove_suffix.

let make_handler_remove_suffix ~content path suffix
    =
  if String.ends_with ~suffix path then
    let alt_path =
      String.sub path 0 (String.length path - String.length suffix)
    in
    [ make_handler ~content alt_path ]
  else []

Updating the website_route definition to use make_handler_remove_suffix is quite easy as well.

 let website_route =
   Dream.scope "~lthms" []
-  @@ List.map
+  @@ List.concat_map
        (fun path ->
          let content = Option.get (Website_content.read path) in
-         make_handler ~content path)
+         if path = "index.html" then
+           (* Special case to deal with "index.html" which needs to be
+              recognized by the route "/" *)
+           [
+             make_handler ~content "/";
+             make_handler ~content "";
+             make_handler ~content "index.html";
+           ]
+         else
+           make_handler_remove_suffix ~content path
+             "/index.html"
+           @ make_handler_remove_suffix ~content
+               path "index.html"
+           @ [ make_handler ~content path ])
        Website_content.file_list

With that, https://soap.coffee/~lthms/posts/index.html returns the same pages as https://soap.coffee/~lthms/posts or https://soap.coffee/~lthms/posts. Check.

Supporting `Content-Type`

Content-Type is an HTTP header which is used by the receiver of the HTTP message (whether it is a request or a response) to interpret its content.

For instance, when building a RPC API, Content-Type is used by the server to know how to parse the request body (Content-Type: application/json or Content-Type: application/octet-stream being two popular choices, for JSON or binary encoding, respectively).

In our case, the Content-Type header is used by the HTTP server to communicate the nature of the content to browsers. For my website, I can just use the file extensions to infer the correct header to set. First, we list the extensions that are actually used.

let content_types =
  [
    (".html", "text/html");
    (".css", "text/css");
    (".xml", "text/xml");
    (".png", "image/png");
    (".svg", "image/svg+xml");
    (".gz", "application/gzip");
    (".pub", "text/plain");
  ]

A header in Dream is encoded as a string * string value, with the first string being the header name and the second being the header value.

let content_type_header path =
  List.filter_map
    (fun (ext, content_type) ->
      if String.ends_with ~suffix:ext path then
        Some ("Content-Type", content_type)
      else None)
    content_types
  |> assert_f
       ~error_msg:Format.(sprintf "Unsupported file type %s" path)
       (( <> ) [])

with assert_f being defined as follows.

let assert_f ~error_msg f v =
  if f v then v else failwith error_msg

assert_f is used to enforce that I don’t deploy a website which contains route lacking a Content-Type header. For instance, if I remove the "html" entry of the content_type list, I get this error when I try to execute the server.

Fatal error: exception Failure("Unsupported file type index.html")

This is because the headers are only computed once, when each route are defined. This is a key principle of this project: compute once, serve many timeI would love to get a compilation error instead (considering there are no runtime values involved), but have not looked into this just yet. .

 let website_route =
   Dream.scope "~lthms" []
   @@ List.concat_map
        (fun path ->
          let content = Option.get (Website_content.read path) in
+         let headers = content_type_header path in
          if path = "index.html" then
            (* Special case to deal with "index.html" which needs to be
               recognized by the route "/" *)
            [
-             make_handler ~content "/";
-             make_handler ~content "";
-             make_handler ~content "index.html";
+             make_handler ~headers ~content "/";
+             make_handler ~headers ~content "";
+             make_handler ~headers ~content "index.html";
            ]
          else
-           make_handler_remove_suffix ~content path
+           make_handler_remove_suffix ~headers ~content path
              "/index.html"
-           @ make_handler_remove_suffix ~content
+           @ make_handler_remove_suffix ~headers ~content
                path "index.html"
-           @ [ make_handler ~content path ])
+           @ [ make_handler ~headers ~content path ])
        Website_content.file_list

(The changes in make_handler and make_handler_remove_prefix are left as an exercise to enthusiast readers)

Compressing if Requested

Nowadays, computations are cheap, while downloading data costs time (and sometimes money). As a consequence, it is often a good idea for a server to compress a large HTTP response, and browsers do ask them to do so, by setting the Accept-Encoding header of their requests.

The value of the Accept-Encoding header is a comma-separated list of supported compression algorithms, optionally ordered with a priority value q.

For instance, Accept-Encoding: gzip;q=0.5, deflate;q=0.3, identity tells you that the browser supports three encoding methods: gzip, deflate and identity (no compression), and the browser prefers gzip over deflate. Besides, the request can provide several Accept-Encoding headers instead of just one, so we can have

Accept-Encoding: gzip;q=0.5
Accept-Encoding: deflate;q=0.3
Accept-Encoding: identity

The String module provides everything we need to check if a browser supports gzip as an encoding methodSpoiler: they do. I was even wondering at some point if I could just always return GZIP-compressed values, ignoring the Accept-Encoding header altogether. If you do that, though, curl becomes annoying to use (it does not uncompress the response automatically, and instead complains about being about to write binary to the standard output). .

(* For [method(; q=val)?], returns [method], except if
   [q=0]. *)
let to_directive str =
  match String.split_on_char ';' str |> List.map String.trim with
  | [ x ] -> Some x
  | [ x; y ] -> (
      match String.split_on_char '=' y |> List.map String.trim with
      | [ "q"; "0" ] -> None
      | [ "q"; _ ] -> Some x
      | _ -> None)
  | _ -> None

(* [contains ~value:v header] returns [true] if [v] is a
   supported method listed in [header]. *)
let contains ~value header =
  String.split_on_char ',' header
  |> List.to_seq |> Seq.map String.trim
  |> Seq.filter_map to_directive
  |> Seq.exists (( = ) value)

We use contains to tell us if we can return a compressed response, which leaves us with one final question: how to compress said response?

The OCaml ecosystem seems to have picked camlzip library when GZIP is involvedYou know it is a legitimate OCaml library when one of the top-level modules is not documented at all . . What is surprising with this library is that it does not support in-memory compression: the functions expect channels, not bytes. That is quite annoying, because we are specifically doing this not to use files.

The Internet is helpful here, and quickly suggests using pipes. It works when you remember –or figure out– that pipes are a blocking mechanism: one does not just write a buffer of arbitrary size in a pipe, because after something like 4KBytes, writing becomes blocking until a read happens to free some space. That’s not a big problem: we can read and write concurrently to the pipe using threads, and OCaml 5 makes it quite easy to do so with the Domain module.

let gzip content =
  let inc, ouc = Unix.pipe () in
  let ouc = Gzip.open_out_chan ~level:6 Unix.(out_channel_of_descr ouc) in
  let _writer =
    Domain.spawn (fun () ->
        Gzip.output_substring ouc content 0 String.(length content);
        Gzip.close_out ouc)
  in
  let res = In_channel.input_all Unix.(in_channel_of_descr inc) in
  Unix.close inc;
  res

Caution

As rightfully pointed out by Daniel Bünzli , the gzip function presented in this article is full of shortcomings. To quote the message, the function can leak fds in case of errors and domains are not meant to be used that way (it’s rather spawn one long running domain per CPU you have). It’s not necessarily more complicated to correct it to use Fun.protect invocations to make sure all your fds get closed even if the function blows up and use Thread.create so that the netizens cut and paste correct code.

In my opinion, this implementation is “good enough” for my use case, which is compressing arbitrary strings before the HTTP server is even started. If it were to be called in the handlers themselves, then definitely, it would not be suitable.

Keep that in mind if you want to borrow this code.

We know how to decide whether to compress or not, and how to compress. The next step is to modify make_handler accordingly.

-let make_handler ~headers ~content path =
+let make_handler ~headers ~gzip_content ~content path =
+  let gzip_headers = ("Content-Encoding", "gzip") :: headers in
   Dream.get path (fun req ->
-    Lwt.return (Dream.response content)))
+      match Dream.headers req "Accept-Encoding" with
+      | accepted_encodings
+        when List.exists (contains ~value:"gzip") accepted_encodings ->
+          Lwt.return @@ Dream.response ~headers:gzip_headers gzip_content
+      | _ -> Lwt.return @@ Dream.response ~headers content)

gzip_content is computed only once (using our gzip function), and passed to make_handler. This way, the only computation the handler needs to do is to “parse” Accept-Encoding.

Caching

Compressing a page to reduce the number of bytes a browser needs to download is fine. Letting the browser know it does not need to download anything because it's previous version is still accurate is better.

This is achieved through two complementary mechanisms: entity tags (ETag)My first encounter with entity tags was around the time GDPR was a hot topic, because you can use them as a cheap replacement for cookies to track your users . I remained at the surface level at the time, it was fun learning more about them through this little project. , and cache policies (Cache-Control).

Entity tags are used to identify a resource, and are expected to change every time the resource is updated. The general workflow goes like this: the first time a browser requests https://soap.coffee/~lthms/index.html, it caches the result along with the value of the ETag header. The next time it needs the page, it adds the header If-None-Match, with the ETag as its value. For such requests, the server is expected to return an empty response with HTTP code 304 (Not Modified).

I decided to use the sha256 hash algorithm to compute the entity tag of each resource of my website. The sha OCaml library looked like a good enough candidate.

(* in `website_route` *)
let etag = Sha256.(string content |> to_hex) in

Interestingly, one question I had to answer was whether the entity tag of a page needed to be different whether it was compressed or not. Internet almost unanimously answered yes. So be it. We just need to keep in mind ETag values are expected to be surrounded by quotes, and we are good to go. It is just a matter of suffixing the ETag with +gzip in the compressed case.

-let make_handler ~headers ~gzip_content ~content path =
-  let gzip_headers = ("Content-Encoding", "gzip") :: headers in
+let make_handler ~headers ~etag ~gzip_content ~content path =
+  let etag_gzip = Format.sprintf "\"%s+gzip\"" etag in
+  let etag = Format.sprintf "\"%s\"" etag in
+  let gzip_headers =
+    ("Content-Encoding", "gzip") :: ("ETag", etag_gzip) :: headers in
+  let identity_headers = ("ETag", etag) :: headers in
   Dream.get path (fun req ->
-      match Dream.headers req "Accept-Encoding" with
-      | accepted_encodings
-        when List.exists (contains ~value:"gzip") accepted_encodings ->
-          Lwt.return @@ Dream.response ~headers:gzip_headers gzip_content
-      | _ -> Lwt.return @@ Dream.response ~headers content)
+      match Dream.headers req "If-None-Match" with
+      | [ previous_etag ] when previous_etag = etag || previous_etag = etag_gzip
+        ->
+          Lwt.return
+          @@ Dream.response
+               ~headers:(("ETag", previous_etag) :: headers)
+               ~code:304 ""
+      | _ -> (
+          match Dream.headers req "Accept-Encoding" with
+          | accepted_encodings
+            when List.exists (contains ~value:"gzip") accepted_encodings ->
+              Lwt.return @@ Dream.response ~headers:gzip_headers gzip_content
+          | _ -> Lwt.return @@ Dream.response ~headers:identity_headers content))

Entity tags are useful, but you still need the browser to make an HTTP request every single time you visit the website. By setting a cache policy, we can remove even remove the need for this request most of the time. The Cache-Control header is used to set a number of parameters, including the max-age value (in seconds).

In my Nginx configuration, I had set max-age to a year for images. I did the same thing here. Besides, I decided to set max-age for other resources to 5 minutes. This seems like a good compromise: since my website does not change very often, it is very unlikely that you happen to visit it when I publish new content. Setting a 5-minute cache policy should let my readers download each resource only once, yet get the freshest version at their next visitDealing with the Cache-Control header is basically the same exercise as setting the correct Content-Type header, and this article is already long enough, which is why there is no diff or snippet in this section. .

And with this, we are done. We get a standalone library to server our website in a browser-friendly manner, which I can theoretically use to replace my current Nginx-powered setup. Although… is it that simple?

Building and Deploying the Website

Files are quite easy to share and deploy. As I mentioned earlier in this article, you just need to rsync them and be done with it. Binaries, on the other hand… One cannot just assume a binary build on a machine X will work on another machine Y. Some additional works need to be done.

The most straightforward solution I know is to rely on static binaries. I have already written about how to generate static binaries for OCaml projects, so I had a pretty strong head start, but one drawback of the approach I’ve described there (and that I have been using for Spatial Shell’s releases Not that there were many of them lately. ) is that it is rather slow (I have a script creating a new local switch each time) and requires static libraries to be installed (which Arch Linux does not provide).

And so, I figured, why not build the static binary in Docker? This allows me to use Alpine to get a static version of my system dependencies, and can be quite fast thanks to Docker build cache . The Dockerfile is quite simple: one stage for building the system and OCaml dependencies, and one stage for building the static binary.

FROM alpine:3.21 AS build_environment

# Use alpine /bin/ash and set shell options
# See https://docs.docker.com/build/building/best-practices/#using-pipes
SHELL ["/bin/ash", "-euo", "pipefail", "-c"]

USER root
WORKDIR /root

RUN apk add autoconf automake bash build-base ca-certificates opam gcc \ 
  git rsync gmp-dev libev-dev openssl-libs-static pkgconf zlib-static \
  openssl-dev zlib-dev
RUN opam init --bare --yes --disable-sandboxing
COPY makefile dune-project .
RUN make _opam/.init OCAML="ocaml-option-static,ocaml-option-no-compression,ocaml.5.2.1"
RUN eval $(opam env) && make server-deps

FROM build_environment AS builder

COPY server ./server
COPY out ./out
COPY dune .
RUN eval $(opam env) && dune build server/main.exe --profile=static

FROM alpine:3.21 AS soap.coffee

COPY --from=builder /root/_build/default/server/main.exe /bin/soap.coffee

Then, building my static binary becomes as simple as:

docker build . -f ./build.Dockerfile \
  --target soap.coffee \
  -t soap.coffee:latest
docker create --name soap-coffee-build soap.coffee:latest
docker cp soap-coffee-build:/bin/soap.coffee .
docker rm -f soap-coffee-build

docker cp does not work on an image, but on a container, so we need to create one which can be destroyed shortly after.

This little binary weights 38MBytes, which seems relatively reasonable to me, considering my website weights 20MBytes. I guess it could be easy to reduce this size by embedding the compressed version of my articles and images, instead of the uncompressed one. But really, for my website, I’m not really interested in investing the extra effort.

Conclusion

I would not recommend anyone to use this in production for anything remotely important, but from my perspective, it was both fun and insightful. I was able to refresh my memories about HTTP “internal,” among other things. Again, as far as I can tell, deploying my website this way did not bring me any benefit, performance-wise; even worse, I am pretty sure the Dream server will not behave as well as Nginx when it comes to handling the load (since it is limited to one core, instead of several with Nginx).

That being said, I am way too invested now… which is why, yes, you are reading a blog post served to you directly from memory 🎉.

On Vestigial Structures

August 16, 2024

On Vestigial Structures

satire ocaml

Do you know what vestigial structures are?This write-up is a slightly modified version of a commit description I have written for $WORK with some help from ChatGPT .

In a nutshell, they are remnants of structures that were functional in an ancestral species but have lost much or all of their original function in the descendant species.

For instance, some snakes, like pythons and boas, have tiny remnants of hind leg bones, which are vestigial structures from when their ancestors had legs. Birds like ostriches and emus have wings that are too small for flight. These wings are vestigial structures as well, remnants of their flying ancestors. Even humans have some, like those small muscles around their ears. These muscles were once used by our ancestors to move their ears, similar to how many animals can today.

Sometimes, vestigial structures are not just useless, they can be harmful. Take our wisdom teeth. They were handy to our ancestors, who had larger jaws and a diet that required more chewing. Sadly, they have become problematic for many modern humans.

Anyway, ocaml-migrate-parsetree is a deprecated OCaml library in $WORK’s dependency tree that will never be compatible with OCaml 5, but it turns out we do not need it anymore .

Building Static Binaries for OCaml Projects

December 31, 2023

Building Static Binaries for OCaml Projects

ocaml

Building static binaries can come in handy. Most notably, when the time comes to distribute executables. I should know, because I spend a bit of time recently preparing for the 6th release of Spatial Shell, and part of that time was ensuring that folks on Linux unwilling to build an OCaml project from source could still give my project a shot.

Turns out, it can be achieved pretty easily, when you know what you are doing. This is not the first article published on the Internet which explains how to build static binaries, but (1) the ones I have found were a bit verbose and hard to follow, and (2) the most recent versions of OCaml bring a new challenge to tackle.

Setting-Up a Dedicated `dune` Profile

At the root of your repository, create a dune file containing the following stanza:

(env
 (static
  (flags
   (:standard -cclib -static))))

This tells dune to passes specific arguments to the OCaml compiler when called with the --profile=static command-line argument.

At this point, calling dune build --profile=static is likely to fail with linking related errors. For instance, I got this one.

/sbin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/crt
beginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
/sbin/ld: failed to set dynamic section sizes: bad value
collect2: error: ld returned 1 exit status

With OCaml 5.1.0 and OCaml 5.1.1, you are also likely to hit an error related to zstd, which is a new runtime dependency for OCaml programsIt looks like a temporary issue. If I understood correctly, the OCaml core developers are planning to remove said dependency in future versions of the compiler. .

/sbin/ld: cannot find -lzstd: No such file or directory
collect2: error: ld returned 1 exit status

The solution for these two problems is the same.

Setting Up a Specific Opam Switch

The OCaml compiler has a lot of so-called variants. The most famous is probably flambda , which enables a series of optimisations for native compilation.

Nowadays, enabling the specific features of a given variant is achieved by installing the related option package when choosing your compiler version. In our case, we have to install:

ocaml-option-static (which will also enables ocaml-option-musl)
ocaml-option-no-compression (for OCaml 5.1.0 and 5.1.1)

The minimal Opam invocation to create such a switch is:

opam switch create . --packages "ocaml-option-static,ocaml-option-no-compression,ocaml.5.1.1"

This particular configuration will produce static binaries using musl , a lightweight libc implementation that is way more suited than the glibc when it comes to static binaries. This means you will have to install musl on your system (it is more than likely that it is packaged by your favorite distribution).

Additionally, you will have to provide the static versions (.a) of the system libraries your project relies on. This part might be tricky to address, and is very dependent on the particular system you are using. For instance, Debian makes this a lot easier than Archlinux. This was typically what was happening with zstd: Archlinux only distributed dynamic libraries, so no .a where available on my laptop. Another example: if your project depends on Zarith , then you have a dependency to gmp. You need to go and find the static library for it before moving to the next sectionOf course, it is also possible that you are working on a pure OCaml project whose only real dependency is to the libc. Lucky you. .

Verifying the Resulting Binaries

Now is the time to call Dune again, with the --profile=static command-line argument. Checking the result of your effort is as simple as calling ldd on the binary compiled by Dune.

    not a dynamic executable

If ldd outputs something of this form, then you are good to go!

Conclusion

This is, in a nutshell, the approach I am using to produce static binaries for Spatial Shell releases. You can find the resulting script in the project repository if you are interested.

If you have spotted an error in this article, or if you think the solution proposed here can be improved, do not hesitate to reach out to me (ideally though my public inbox). Or, even better, open an issue on Spatial Shell tracker ! Afterall, these are the very steps I’m relying on in order to provide static binaries, so I’d like to know if I am doing something wrong 😅.

Extensible Records in OCaml Using dmap

June 20, 2023

Extensible Records in OCaml Using `dmap`

ocaml

dmap is a library to create and manipulate heterogeneous maps. It features a very straightforward API which leverages the common trick of tagging the type of the keys with a parameter specifying the types of the associated valuesThis article assumes readers are familiar with GADTs. . That is, given 'a key the type of keys of a heterogeneous map t, then the value associated to a key k of type bool key is expected to be of type bool.

In this write-up, we show how it is possible to use dmap to implement extensible records (that is, records which can be extended with new fields after they have been defined) in OCaml. I have also published on GitHub a repository containing the implementation presented in this article . While dmap is far from being the only available solution for this use caseFor heterogeneous maps, gmap is apparently a popular alternative, and so is hmap . For extensible records, orec looks like a very interesting library I should probably take a long look at. , it is the one I happened to use for implementing a “plugin system” in a library where each plugin can manipulate a shared state in a type-safe way.

Encoding Extensible Records

The (simplified) module type of a heterogeneous map built using dmap is as follows.

module type S = sig
  type t
  
  type 'a key

  type binding = Binding : 'a key * 'a -> binding

  val empty : t

  val add : 'a key -> 'a -> t -> t

  val find_opt : 'a key -> t -> 'a option

  val fold_left : ('a -> binding -> a) -> 'a -> t -> 'a
end

The use case for heterogeneous maps I’ve been interested in recently is encoding records. More precisely, constructors of keys encode fields containing collections of values. In such approach, a constructor Foo : int key encodes a field of type int option, while a constructor Bar : string -> bool key encodes a field of type bool StringMap.t. That is, the heterogeneous map HMap.t, such as

type _ key = Foo : int key | Bar : string -> bool key

module HMap : S with type 'a key  = 'a key

can be used to encode values of a record type

type t = {
  foo : int option;
  bar : bool StringMap.t;
}

as the following conversion functions demonstrates.

let of_record {foo; bar} =
  let res = HMap.empty in
  let res = HMap.add Foo foo res in
  StringMap.fold_left
    (fun res (key, value) -> HMap.add (Bar key) value res)
    res
    bar

let of_hmap map =
  {
    foo = HMap.find_opt Foo map;
    bar = HMap.fold_left
            (fun smap (Binding binding) ->
               match binding with
               | (Bar k, v) -> StringMap.add k v smap
               | _ -> smap)
            StringMap.empty
            map;
  }

As a consequence, to construct a heterogeneous map whose type of keys is extensible is a way to get extensible records in OCaml.

To that end, dmap only asks for a function to compare two arbitrary keys. More precisely, given the type

type _ key = ..

the challenge is to write the function compare of type

val compare : 'a key -> 'b key -> ('a, 'b) Dmap.cmp

with Dmap.cmp being definedThis definition of cmp enforces that two keys can only be equal if and only if the values associated to them are of the same type. as

type (_, _) cmp =
  | Lt : ('a, 'b) cmp
  | Eq  : ('a, 'a) cmp
  | Gt : ('a, 'b) cmp

The Solution

To use extensible types in this kind of setting, the trick I am aware of is to use a registration mechanismFor instance, this is what the tezos-error-monad package is using for being able to print the values of its extensible error type . .

The compare function will work as follows.

We assign an integer to each constructor of the type key.
We compare these integers when the arguments of compare are constructed from different constructors.
We implement one comparison function per constructors, to be called when the two arguments of compare are the same.

The exercise is being made a bit more complicated than what the previous paragraph suggests due to the fact that both key and cmp are GADTs. As a consequence, the rest of the article will proceed as follows. First, we will implement the overall logic of the compare function implemented with a registration mechanism, and only then will we tackle the OCaml compiler errors GADTs so often bring.

Step 1: Forget About GADTs

To implement our registration mechanism, we rely on a good old reference to a listIt is a good idea to hide this implementation “details” behind a nice API, using a mli file. . The position of a registered object within the list will determine the integer attributed to it.

let registered_keys = ref []

let register_key key =
  registered_keys := key :: !registered_keys

In this list, we store a collection of first-class modules which provide us everything we need to compare two 'a key values obtained from the same constructors.

module type KEY = sig
  type t

  val proj : 'a key -> t option
  val compare : t -> t -> int
end

The type t of the module type KEY is the type of one constructor’s arguments. The simplest definition of t is unit, which means the field encoded with the related constructor points to a singleton value (as foo did in the previous section).

However, it is also possible to have keys’ constructors taking arguments, which would translate into a field pointing to collections of value. This was the case for bar, which was pointing to a mapping of strings to booleans.

The compare function will consist in iterating over the first-class modules registered in registered_keys, in order to determine to which ones the two arguments belong to.

This means that considering a call compare left right, for each first-class module of registered_keys, either

both left and right have not yet been associated to a first-class module,
or left has, but `right hasn’t
or right has, but left hasn’t
or both left and `right have been associated to a first-class module.

Only when the fourth stage has been reached can we provide the expected result of compare, whose type we have simplified in this section to return a int instead of the GADT Dmap.cmp.

To encode these stages, we introduce a dedicated accumulator type.

type ('a, 'b) acc =
  | Init : 'a key * 'b key -> ('a, 'b) acc
  | Compare_left_with : 'a key * int -> ('a, 'b) acc
  | Compare_right_with : int * 'b key -> ('a, 'b) acc
  | Res : int -> ('a, 'b) acc

This allows us to implement compareNote the use of the Seq module here. It is motivated by the fact that List.fold_lefti does not exist. We need to know the position of the current first-class module we are testing the keys against, hence the need for a fold_lefti. Of course, we could have added the current position to the acc type, but the function is already cumbersome enough, and using Seq.t is mostly free. .

let compare : type a b. a key -> b key -> int =
 fun left right ->
  List.to_seq !registered_keys
  |> Seq.fold_lefti
       (fun (acc : (a, b) acc) i (module K : KEY) ->
         match acc with
         | Init (left, right) -> (
             match (K.proj left, K.proj right) with
             | Some left, Some right -> Res (K.compare left right)
             | Some _, None -> Compare_right_with (i, right)
             | None, Some _ -> Compare_left_with (left, i)
             | None, None -> acc)
         | Compare_right_with (j, right) -> (
             match K.proj right with
             | Some _ -> Res (Int.compare j i)
             | None -> acc)
         | Compare_left_with (left, j) -> (
             match K.proj left with
             | Some _ -> Res (Int.compare i j)
             | None -> acc)
         | _ -> acc)
       (Init (left, right))
  |> function
  | Res x -> x
  | _ ->
      raise
        (Invalid_argument
           "comparison with at least one unregistered key variant")

This function gives us the overall structure and logic of compare, but of course we are not done yet.

Step 2: Deal with GADTs

In practice and as hinted in the introduction of this section, dmap is not expected a comparison function of type 'a key -> 'b key -> ('a, 'b) Dmap.cmp, not 'a key -> 'b key -> int.

The first step is to update acc accordingly.

type ('a, 'b) acc =
   | Init : 'a key * 'b key -> ('a, 'b) acc
   | Compare_left_with : 'a key * int -> ('a, 'b) acc
   | Compare_right_with : int * 'b key -> ('a, 'b) acc
-  | Res : int -> ('a, 'b) acc
+  | Res : ('a, 'b) Dmap.cmp -> ('a, 'b) acc

Then, we focus our attention to modifying compare, by modifying its type signature, and by updating its definition to make use of the new acc type.

-let compare : type a b. a key -> b key -> int =
+let compare : type a b. a key -> b key -> (a, b) Dmap.cmp =
  fun left right ->
   List.to_seq !registered_keys
   |> Seq.fold_lefti
       (fun (acc : (a, b) acc) i (module K : KEY) ->
          match acc with
          | Init (left, right) -> (
              match (K.proj left, K.proj right) with
-             | Some left, Some right -> Res (K.compare left right)
+             | Some left, Some right ->
+                 let x = K.compare left right in
+                 Res (if x = 0 then Eq else if x < 0 then Lt else Gt)
              | Some _, None -> Compare_right_with (i, right)
              | None, Some _ -> Compare_left_with (left, i)
              | None, None -> acc)
          | Compare_right_with (j, right) -> (
              match K.proj right with
-             | Some _ -> Res (Int.compare j i)
+             | Some _ ->
+                 let x = Int.compare j i in
+                 Res (if x < 0 then Lt else Gt)
              | None -> acc)
          | Compare_left_with (left, j) -> (
              match K.proj left with
-             | Some _ -> Res (Int.compare i j)
+             | Some _ ->
+                 let x = Int.compare j i in
+                 Res (if x < 0 then Lt else Gt)
              | None -> acc)
          | _ -> acc)
        (Init (left, right))

At this point, we are remembered that the OCaml compiler is not that easy to please when GADTs are involved. In particular, it complains for our use of Eq.

46 |               Res (if x = 0 then Eq else if x < 0 then Lt else Gt)
                                      ^^
Error: This expression has type (a, a) Dmap.cmp
       but an expression was expected of type (a, b) Dmap.cmp
       Type a is not compatible with type b

This error message is the result of a tragedy in two acts:

The Eq constructor of cmp is of type ('a, 'a) cmp. That is, only two keys associated with values of the same type can be equal.
compare takes two keys whose types can be different, and even if proj is expected to return a Some only when both keys are constructed with the same constructor, nothing in its type enforces that invariant.

As such, OCaml is not convinced with our code alone that a = b, and rejects it.

One possible trick here is to rely on an equality witness typeThis is the trick to keep in mind when playing with GADTs, to a point where eq could arguably be added to the OCaml’s standard library. , and refine proj in order to convince OCaml our code is type-safe.

type (_, _) eq = Refl : ('a, 'a) eq

We then modify KEY to have proj not only return the arguments of the constructor, but also a type equality witness to unify the polymorphic argument of key to the expected value type of said constructor.

 module type KEY = sig
   type t
+  type r
 
-  val proj : 'a key -> t option
+  val proj : 'a key -> (t * ('a, r) eq) option
   val compare : t -> t -> int
 end

This is enough to convince OCaml that a = b, because it gets a proof that a = K.r and b = K.rBut only if you actually match Refl explicitly, meaning the pattern Some (left, _), Some (right, _) would still lead to the same error as not returning Refl at all. .

           match (K.proj left, K.proj right) with
-          | Some left, Some right ->
+          | Some (left, Refl), Some (right, Refl) ->
               let x = K.compare left right in
               Res (if x = 0 then Eq else if x < 0 then Lt else Gt)
           | Some _, None -> Compare_right_with (i, right)
           | None, Some _ -> Compare_left_with (left, i)

That’s it. compare is done, compiles, and satisfies the expectations of dmap. We can instantiate its functor with our key extensible typeThis approach will lead every value of type Extensible_record.t to share the same fields, which might not be you need. One can imagine embedding the registered_keys list of known keys inside the extensible records values, along with a heterogeneous map. This would allow each value to have its own dynamic set of fields. .

module Extensible_record = Dmap.Make (struct
  type 'a t = 'a key

  let compare = compare
end)

Step 3: Profit

To wrap up this article, we can come back to our initial example of a record with two fields foo and bar.

Here is how we can extend key to have a Foo constructor.

type _ key += Foo : int key

let () =
  register_key
    (module struct
      type t = unit
      type r = int

      let proj : type a. a key -> (t * (a, r) eq) option = function
        | Foo -> Some ((), Refl)
        | _ -> None

      let compare () () = 0
    end)

Similarly, here is how we can register Bar.

type _ key += Bar : string -> bool key

let () =
  register_key
    (module struct
      type t = string
      type r = bool

      let proj : type a. a key -> (t * (a, r) eq) option = function
        | Bar arg -> Some (arg, Refl)
        | _ -> None

      let compare = String.compare
    end)

Once registered, these two fields/keys can be used to populate an Extensible_record.t value with values of the expected types.

let () =
  let record = Extensible_record.empty in
  let record = Extensible_record.add Foo 3 record in
  let record = Extensible_record.add (Bar "foobar") true record in
  assert (Extensible_record.find Foo record = 3) ;
  assert (Extensible_record.find (Bar "foobar") record)

A nice happy consequence of this approach is that it allows us to register private fields only a given module can manipulate. Indeed, if this module does not expose the constructor of the key it relies on, then only it can interact with this part of the extensible record.

Neovim, OCaml Interfaces, Tree-Sitter and LSP

May 1, 2023

Neovim, OCaml Interfaces, Tree-Sitter and LSP

ocaml neovim

Can we all agree that witnessing syntax highlighting being absolutely off is probably the most annoying thing that can happen to anybody?

I mean, just look at this horror.

Syntax highlighting being absolutely wrong.

What you are looking at is the result of trying to enable tree-sitter for OCaml hacking and calling it a day. In a nutshell, OCaml mli files are quickly turning into a random mess of nonsensical colors, and I didn’t know why. I tried to blame tree-sitter-ocaml , but, of course I was wrong.

The issue is subtle, and to be honest, I don’t know if I totally grasp it. But from my rough understanding, it breaks down as follows.

tree-sitter-ocaml defines two grammars: ocaml for the ml files, and ocaml_interface (but ocamlinterface also works) for the mli files
By default, neovim uses the filetype ocaml for mli files, so the incorrect parser is being used for syntax highlighting. This explains the root issue
Bonus: ocamllsp does not recognize the ocamlinterface filetype by default (but somehow use the ocaml.interface id for mli files…There is probably something to be done here. ).

So, in order to have both tree-sitter and ocamllsp working at the same time, I had to tweak my configuration a little bit.

lspconfig.ocamllsp.setup({
  filetypes = vim.list_extend(
    require('lspconfig.server_configurations.ocamllsp')
      .default_config
      .filetypes,
    { 'ocamlinterface' }
  ),
})

vim.cmd([[au! BufNewFile,BufRead *.mli setfiletype ocamlinterface]])

And now, I am blessed with a consistent syntax highlighting for my mli files.

Syntax highlighting being absolutely right.

Writing a Function Whose Argument is a Polymorphic Function in OCaml

August 7, 2022

Writing a Function Whose Argument is a Polymorphic Function in OCaml

ocaml

In OCaml, it is not possible to write a function whose argument is a polymorphic function. Trying to write such a function results in the type-checker complaining back at you.

let foo (type a b) id (x : a) (y : b) = (id x, id y)

Line 1, characters 50-51:
1 | let foo (type a b) id (x : a) (y : b) = (id x, id y);;
                                                      ^
Error: This expression has type b but an expression was expected
of type a

When OCaml tries to type check foo, it infers id expects an argument of type a because of id x, then fails when trying to type check id y.

The trick to be able to write foo is to use records. Indeed, while the argument of a function cannot be polymorphic, the field of a record can. This effectively makes it possible to write foo, at the cost of a level of indirection.

type id = {id : 'a. 'a -> 'a}

let foo {id} x y = (id x, id y)

From a runtime perspective, it is possible to tell OCaml to remove the introduced indirection with the unboxed annotation. There is nothing we can do in the source, though. We need to destruct id in foo, and we need to construct it at its call site.

g {id = fun x -> x}

As a consequence, this solution is not a silver bullet, but it is an option that is worth considering if, e.g., it allows us to export a cleaner API to the consumer of a module. Personally, I have been considering this trick recently to remove the need for a library to be implemented as a functor.

Implementing an Echo Server in Coq with coqffi.1.0.0

December 10, 2020

Implementing an Echo Server in Coq with `coqffi.1.0.0`

coq ocaml coqffi

In this article, we will demonstrate how coqffi can be used to implement an echo server, i.e., a TCP server which sends back any input it receives from its clients. In addition to coqffi, you will need to install coq-simple-io. The latter is available in the released repository of the Opam Coq Archive .

opam install coq-coqffi coq-simple-io

Besides, you can download the source tree presented in this article if you want to try to read the source directly, or modify it to your taste.

Project Layout

Before diving too much into the implementation of our echo server, we first give an overview of the resulting project’s layout. Since we aim at implementing a program, we draw our inspiration from the idiomatic way of organizing a OCaml project.

We have three directories at the root of the project.

ffi/ contains the low-level OCaml code: It provides an OCaml library (ffi), and a Coq theory (FFI) which gathers the FFI modules generated by coqffi.
src/ contains the Coq implementation of our echo server: It provides a Coq theory (Echo) which depends on the FFI theory the SimpleIO theory of coq-simple~io. This theory provides the implementation of our echo server in Coq.
bin/ contains the pieces of code to get an executable program: It contains a Coq module (echo.v) which configures and uses the extraction mechanism to generate an OCaml module (echo.ml). This OCaml module can be compiled to get an executable program.

Note that we could have decided to only have one Coq theory. We could also have added a fourth directory (theories/) for formal verification specific code, but this is out of the scope of this tutorial.

Overall, we use dune to compile and compose the different parts of the echo server. dune has a native —yet unstable at the time of writing— support for building Coq projects, with very convenient stanzas like coq.theory and coq.extraction.

The following graph summarizes the dependencies between each component (plain arrows symbolize software dependencies).

The echo server dependency graph. Dashed boxes are generated.

We enable Coq-related stanza with (using coq 0.2) in the dune-project. file.

(lang dune 2.7)
(using coq 0.2)

The rest of this tutorial proceeds by diving into each directory.

FFI Bindings

Our objective is to implement an echo server, i.e., a server which (1) accepts incoming connections, and (2) sends back any incoming messages. We will consider two classes of effects. One is related to creating and manipulating TCP sockets. The other is dedicated to process management, more precisely to be able to fork when receiving incoming connections.

Therefore, the ffi library will provide two modules. Likewise, the FFI theory will provide two analogous modules generated by coqffi.

In the ffi/ directory, we add the following stanza to the dune file.

(library
  (name ffi)
  (libraries unix))

dune will look for any .ml and .mli files within the directory and will consider they belong to the ffi library. We use the unix library to implement the features we are looking for.

Then, we add the following stanza to the dune file of the ffi/ directory.

(coq.theory
  (name FFI))

This tells dune to look for .v file within the ffi/ directory, in order to build them with Coq. A nice feature of dune is that if we automatically generate Coq files, they will be automatically “attached” to this theory.

Sockets

Sockets are boring. The following OCaml module interface provides the necessary type and functions to manipulate them.

type socket_descr

val open_socket : string -> int -> socket_descr
val listen : socket_descr -> unit
val recv : socket_descr -> string
val send : socket_descr -> string -> int
val accept_connection : socket_descr -> socket_descr
val close_socket : socket_descr -> unit

Our focus is how to write the interface modules for coqffi. Since the object of this tutorial is not the implementation of an echo server in itself, the implementation details of the ffi library will not be discussed, but is provided at the end of this article.

dune generates .cmi files for the .mli files of our library, and provides the necessary bits to easily locate them. Besides, the action stanza can be used here to tell to dune how to generate the module Socket.v from file.cmi. We add the following entry to ffi/dune.

(rule
  (target Socket.v)
  (action (run coqffi %{cmi:socket} -o %{target})))

We call coqffi without any feature-related command-line argument, which means only the simple-io feature is enabled. As a consequence, the socket_descr type is axiomatized in Coq, and in addition to a MonadSocket monad, coqffi will generate an instance for this monad for the IO monad of coq-simple-io.

The stanza generates the following Coq module.

(* This file has been generated by coqffi. *)

Set Implicit Arguments.
Unset Strict Implicit.
Set Contextual Implicit.
Generalizable All Variables.
Close Scope nat_scope.

From CoqFFI Require Export Extraction.
From SimpleIO Require Import IO_Monad.

Axiom socket_descr : Type.

Extract Constant socket_descr => "Ffi.Socket.socket_descr".

(** * Impure Primitives *)

(** ** Monad Definition *)

Class MonadSocket (m : Type -> Type) : Type :=
  { open_socket : string -> i63 -> m socket_descr
  ; listen : socket_descr -> m unit
  ; recv : socket_descr -> m string
  ; send : socket_descr -> string -> m i63
  ; accept_connection : socket_descr -> m socket_descr
  ; close_socket : socket_descr -> m unit
  }.

(** ** [IO] Instance *)

Axiom io_open_socket : string -> i63 -> IO socket_descr.
Axiom io_listen : socket_descr -> IO unit.
Axiom io_recv : socket_descr -> IO string.
Axiom io_send : socket_descr -> string -> IO i63.
Axiom io_accept_connection : socket_descr -> IO socket_descr.
Axiom io_close_socket : socket_descr -> IO unit.

Extract Constant io_open_socket
  => "(fun x1 x2 k__ -> k__ ((Ffi.Socket.open_socket x1 x2)))".
Extract Constant io_listen => "(fun x1 k__ -> k__ ((Ffi.Socket.listen x1)))".
Extract Constant io_recv => "(fun x1 k__ -> k__ ((Ffi.Socket.recv x1)))".
Extract Constant io_send
  => "(fun x1 x2 k__ -> k__ ((Ffi.Socket.send x1 x2)))".
Extract Constant io_accept_connection
  => "(fun x1 k__ -> k__ ((Ffi.Socket.accept_connection x1)))".
Extract Constant io_close_socket
  => "(fun x1 k__ -> k__ ((Ffi.Socket.close_socket x1)))".

Instance IO_MonadSocket : MonadSocket IO :=
  { open_socket := io_open_socket
  ; listen := io_listen
  ; recv := io_recv
  ; send := io_send
  ; accept_connection := io_accept_connection
  ; close_socket := io_close_socket
  }.

(* The generated file ends here. *)

Process Management

In order to avoid a client to block the server by connecting to it without sending anything, we can fork a new process for each client.

type identity = Parent of int | Child

val fork : unit -> identity

This time, the proc.mli module interface introduces a transparent type, /i.e./, it also provides its definition. This is a good use case for the transparent-types feature of coqffi. In the stanza for generating Proc.v, we enable it with the -ftransparent-types command-line argument, like this.

(rule
  (target Proc.v)
  (action (run coqffi -ftransparent-types %{cmi:proc} -o %{target})))

which generates the following Coq module.

(* This file has been generated by coqffi. *)

Set Implicit Arguments.
Unset Strict Implicit.
Set Contextual Implicit.
Generalizable All Variables.
Close Scope nat_scope.

From CoqFFI Require Export Extraction.
From SimpleIO Require Import IO_Monad.

Inductive identity : Type :=
| Parent (x0 : i63) : identity
| Child : identity.

Extract Inductive identity => "Ffi.Proc.identity"
  [ "Ffi.Proc.Parent" "Ffi.Proc.Child" ].

(** * Impure Primitives *)

(** ** Monad Definition *)

Class MonadProc (m : Type -> Type) : Type := { fork : unit -> m identity
                                             }.

(** ** [IO] Instance *)

Axiom io_fork : unit -> IO identity.

Extract Constant io_fork => "(fun x1 k__ -> k__ ((Ffi.Proc.fork x1)))".

Instance IO_MonadProc : MonadProc IO := { fork := io_fork
                                        }.

(* The generated file ends here. *)

We now have everything we need to implement an echo server in Coq.

Implementing an Echo Server

Our implementation will be part of a dedicated Coq theory, called Echo. This is done easily a dune file in the src/ directory, with the following content.

(coq.theory
  (name Echo)
  (theories FFI))

In the rest of this section, we will discuss the content of the unique module of this theory. Hopefully, readers familiar with programming impurity by means of monads will not find anything particularly surprising here.

Let us start with the inevitable sequence of import commands. We use the Monad and MonadFix typeclasses of ExtLib, and our FFI modules from the FFI theory we have previously defined.

From ExtLib Require Import Monad MonadFix.
From FFI Require Import Proc Socket.

Letting Coq guess the type of unintroduced variables using the ` annotation (e.g., in presence of `{Monad m}, Coq understands m is of type Type -> Type) is always nice, so we enable it.

Generalizable All Variables.

We enable the monad notation provided by ExtLib. In this article, we prefer the let* notation (as recently introduced by OCaml) over the <- notation of Haskell, but both are available.

Import MonadLetNotation.
Open Scope monad_scope.

Then, we define a notation to be able to define local, monadic recursive functions using the mfix combinator of the MonadFix typeclass.

Notation "'let_rec*' f x ':`' p 'in' q" :`
  (let f :` mfix (fun f x `> p) in q)
    (at level 61, x pattern, f name, q at next level, right associativity).

Note that mfix does /not/ check whether or not the defined function will terminate (contrary to the fix keyword of Coq). This is fortunate because in our case, we do not want our echo server to converge, but rather to accept an infinite number of connections.

We can demonstrate how this notation can be leveraged by defining a generic TCP server, parameterized by a handler to deal with incoming connections.

Definition tcp_srv `{Monad m, MonadFix m, MonadProc m, MonadSocket m}
    (handler : socket_descr -> m unit)
  : m unit :=
  let* srv := open_socket "127.0.0.1" 8888 in
  listen srv;;

  let_rec* tcp_aux _ :=
    let* client := accept_connection srv in
    let* res := fork tt in
    match res with
    | Parent _ => close_socket client >>= tcp_aux
    | Child =>  handler client
    end
  in

  tcp_aux tt.

The handler for the echo server is straightforward: it just reads incoming bytes from the socket, sends it back, and closes the socket.

Definition echo_handler `{Monad m, MonadSocket m} (sock : socket_descr)
  : m unit :=
  let* msg := recv sock in
  send sock msg;;
  close_socket sock.

Composing our generic TCP server with our echo handler gives us an echo server.

Definition echo_server `{Monad m, MonadFix m, MonadProc m, MonadSocket m}
  : m unit :=
  tcp_srv echo_handler.

Because coqffi has generated typeclasses for the impure primitives of proc.mli and socket.mli, echo_server is polymorphic, and can be instantiated for different monads. When it comes to extracting our program, we will generally prefer the IO monad of coq-simple-io. But we could also imagine verifying the client handler with FreeSpec, or the generic TCP server with Interaction Trees (which support diverging computations). Overall, we can have different verification strategies for different parts of our program, by leveraging the most relevant framework for each part, yet being able to extract it in an efficient form.

The next section shows how this last part is achieved using, once again, a convenient stanza of dune.

Extracting and Building an Executable

The 0.2 version of the Coq-related stanzas of dune provides the coq.extraction stanza, which can be used to build a Coq module expected to generate ml files.

In our case, we will write bin/echo.v to extract the echo_server in a echo.ml module, and uses the executable stanza of dune to get an executable from this file. To achieve this, the bin/dune file simply requires these two stanzas.

(coq.extraction
  (prelude echo)
  (theories Echo)
  (extracted_modules echo))

(executable
  (name echo)
  (libraries ffi))

We are almost done. We now need to write the echo.v module, which mostly consists of (1) providing a MonadFix instance for the IO monad, (2) using the IO.unsafe_run function to escape the IO monad, (3) calling the Extraction command to wrap it up.

From Coq Require Extraction.
From ExtLib Require Import MonadFix.
From SimpleIO Require Import SimpleIO.
From Echo Require Import Server.

Instance MonadFix_IO : MonadFix IO :=
  { mfix := @IO.fix_io }.

Definition main : io_unit :=
  IO.unsafe_run echo_server.

Extraction "echo.ml" main.

Since we are using the i63 type (signed 63bits integers) of the CoqFFI theory, and since i63 is implemented under the hood with Coq primitive integers, we also need to provide a Uint63 module with a of_int function. Fortunately, this module is straightforward to write.

let of_int x = x

And voilà. A call to dune at the root of the repository will build everything (Coq and OCaml alike). Starting the echo server is as simple as

dune exec bin/echo.exe

And connecting to it can be achieved with a program like telnet.

$ telnet 127.0.0.1 8888
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
hello, echo server!
hello, echo server!
Connection closed by foreign host.

Appendix

The `Socket` OCaml Module

There is not much to say, except that (as already stated) we use the Unix module to manipulate sockets, and we attach to each socket a buffer to store incoming bytes.

let buffer_size = 1024

type socket_descr = {
  fd : Unix.file_descr;
  recv_buffer : bytes;
}

let from_fd fd =
  let rbuff = Bytes.create buffer_size in
  { fd ` fd; recv_buffer ` rbuff }

let open_socket hostname port =
  let open Unix in
  let addr = inet_addr_of_string hostname in
  let fd = socket PF_INET SOCK_STREAM 0 in
  setsockopt fd SO_REUSEADDR true;
  bind fd (ADDR_INET (addr, port));
  from_fd fd

let listen sock = Unix.listen sock.fd 1

let recv sock =
  let s = Unix.read sock.fd sock.recv_buffer 0 buffer_size in
  Bytes.sub_string sock.recv_buffer 0 s

let send sock msg =
  Unix.write_substring sock.fd msg 0 (String.length msg)

let accept_connection sock =
  Unix.accept sock.fd |> fst |> from_fd

let close_socket sock = Unix.close sock.fd

The `Proc` OCaml Module

Thanks to the Unix module, the implementation is pretty straightforward.

type identity = Parent of int | Child

let fork x =
  match Unix.fork x with
  | 0 -> Child
  | x -> Parent x

coqffi.1.0.0 In A Nutshell

December 10, 2020

`coqffi.1.0.0` In A Nutshell

coq ocaml coqffi

For each entry of a cmi file (a compiled mli file), coqffi tries to generate an equivalent (from the extraction mechanism perspective) Coq definition. In this article, we walk through how coqffi works.

Note that we do not dive into the vernacular commands coqffi generates. They are of no concern for users of coqffi.

Getting Started

Requirements

The latest version of coqffi (1.0.0~beta8) is compatible with OCaml 4.08 up to 4.14, and Coq 8.12 up top 8.13. If you want to use coqffi, but have incompatible requirements of your own, feel free to submit an issue .

Installing `coqffi`

The recommended way to install coqffi is through the Opam Coq Archive , in the released repository. If you haven’t activated this repository yet, you can use the following bash command.

opam repo add coq-released https://coq.inria.fr/opam/released

Then, installing coqffi is as simple as

opam install coq-coqffi

You can also get the source from the upstream git repository . The README provides the necessary pieces of information to build it from source.

Additional Dependencies

One major difference between Coq and OCaml is that the former is pure, while the latter is not. Impurity can be modeled in pure languages, and Coq does not lack of frameworks in this respect. coqffi currently supports two of them: coq-simple-io and FreeSpec . It is also possible to use it with Interaction Trees , albeit in a less direct manner.

Primitive Types

coqffi supports a set of primitive types, i.e., a set of OCaml types for which it knows an equivalent type in Coq. The list is the following (the Coq types are fully qualified in the table, but not in the generated Coq module as the necessary Import statements are generated too).

OCaml type	Coq type
`bool`	`Coq.Init.Datatypes.bool`
`char`	`Coq.Strings.Ascii.ascii`
`int`	`CoqFFI.Data.Int.i63`
`'a list`	`Coq.Init.Datatypes.list a`
`'a Seq.t`	`CoqFFI.Data.Seq.t`
`'a option`	`Coq.Init.Datatypes.option a`
`('a, 'e) result`	`Coq.Init.Datatypes.sum`
`string`	`Coq.Strings.String.string`
`unit`	`Coq.Init.Datatypes.unit`
`exn`	`CoqFFI.Exn`

The i63 type is introduced by the CoqFFI theory to provide signed primitive integers to Coq users. They are implemented on top of the (unsigned) Coq native integers introduced in Coq 8.13. The i63 type will be deprecated once the support for signed primitive integers is implementedThis is actually one of the sources of incompatibility of coqffi with most recent versions of Coq. .

When processing the entries of a given interface model, coqffi will check that they only use these types, or types introduced by the interface module itself.

Sometimes, you may encounter a situation where you have two interface modules b.mli and b.mli, such that b.mli uses a type introduced in a.mli. To deal with this scenario, you can use the --witness flag to generate A.v. This will tell coqffi to also generate A.ffi; this file can then be used when generating B.v thanks to the -I option. Furthermore, for B.v to compile the --require option needs to be used to ensure the A Coq library (A.v) is required.

To give a more concrete example, given ~a.mli~

type t

and b.mli

type a = A.t

To generate A.v, we can use the following commands:

ocamlc a.mli
coqffi --witness -o A.v a.cmi

Which would generate the following axiom for t.

Axiom t : Type.

Then, generating B.v can be achieved as follows:

ocamlc b.mli
coqffi -I A.ffi -ftransparent-types -r A -o B.v b.cmi

which results in the following output for v:

Require A.

Definition u : Type := A.t.

Code Generation

coqffi distinguishes five types of entries: types, pure values, impure primitives, asynchronous primitives, exceptions, and modules. We now discuss how each one of them is handled.

Types

By default, coqffi generates axiomatized definitions for each type defined in a .cmi file. This means that type t becomes Axiom t : Type. Polymorphism is supported, i.e., type 'a t becomes Axiom t : forall (a : Type), Type.

It is possible to provide a “model” for a type using the coq_model annotation, for instance, for reasoning purposes. That is, we can specify that a type is equivalent to a list.

type 'a t [@@coq_model "list"]

This generates the following Coq definition.

Definition t : forall (a : Type), Type := list.

It is important to be careful when using the =coq_model= annotation. More precisely, the fact that t is a list in the “Coq universe” shall not be used while the implementation phase, only the verification phase.

Unnamed polymorphic type parameters are also supported. In presence of such parameters, coqffi will find it a name that is not already used. For instance,

type (_, 'a) ast

becomes

Axiom ast : forall (b : Type) (a : Type), Type.

Finally, coqffi has got an experimental feature called transparent-types (enabled by using the -ftransparent-types command-line argument). If the type definition is given in the module interface, then coqffi tries to generate an equivalent definition in Coq. For instance,

type 'a llist =
  | LCons of 'a * (unit -> 'a llist)
  | LNil

becomes

Inductive llist (a : Type) : Type :=
| LCons (x0 : a) (x1 : unit -> llist a) : llist a
| LNil : llist a.

Mutually recursive types are supported, so

type even = Zero | ESucc of odd
and odd = OSucc of even

becomes

Inductive odd : Type :=
| OSucc (x0 : even) : odd
with even : Type :=
| Zero : even
| ESucc (x0 : odd) : even.

Besides, coqffi supports alias types, as suggested in this write-up when we discuss witness files.

The transparent-types feature is experimental, and is currently limited to variant types. It notably does not support records. Besides, it may generate incorrect Coq types, because it does not check whether or not the positivity condition is satisfied.

Pure values

coqffi decides whether or not a given OCaml value is pure or impure with the following heuristics:

Constants are pure
Functions are impure by default
Functions with a coq_model annotation are pure
Functions marked with the pure annotation are pure
If the pure-module feature is enabled (-fpure-module), then synchronous functions (which do not live inside the ~Lwt~ monad) are pure

Similarly to types, coqffi generates axioms (or definitions if the coq_model annotation is used) for pure values. Then,

val unpack : string -> (char * string) option [@@pure]

becomes

Axiom unpack : string -> option (ascii * string).

Polymorphic values are supported.

val map : ('a -> 'b) -> 'a list -> 'b list [@@pure]

becomes

Axiom map : forall (a : Type) (b : Type), (a -> b) -> list a -> list b.

Again, unnamed polymorphic typse are supported, so

val ast_to_string : _ ast -> string [@@pure]

becomes

Axiom ast_to_string : forall (a : Type), string.

Impure Primitives

coqffi reserves a special treatment for /impure/ OCaml functions. Impurity is usually handled in pure programming languages by means of monads, and coqffi is no exception to the rule.

Given the set of impure primitives declared in an interface module, coqffi will (1) generate a typeclass which gathers these primitives, and (2) generate instances of this typeclass for supported backends.

We illustrate the rest of this section with the following impure primitives.

val echo : string -> unit
val scan : unit -> string

where echo allows writing something the standard output, and scan to read the standard input.

Assuming the processed module interface is named console.mli, the following Coq typeclass is generated.

Class MonadConsole (m : Type -> Type) := { echo : string -> m unit
                                         ; scan : unit -> m string
                                         }.

Using this typeclass and with the additional support of an additional Monad typeclass, we can specify impure computations which interacts with the console. For instance, with the support of ExtLib, one can write.

Definition pipe `{Monad m, MonadConsole m} : m unit :=
  let* msg := scan () in
  echo msg.

There is no canonical way to model impurity in Coq, but over the years several frameworks have been released to tackle this challenge.

coqffi provides three features related to impure primitives.

`simple-io`

When this feature is enabled, coqffi generates an instance of the typeclass for the =IO= monad introduced in the coq-simple-io package

Axiom io_echo : string -> IO unit.
Axiom io_scan : unit -> IO string.

Instance IO_MonadConsole : MonadConsole IO := { echo := io_echo
                                              ; scan := io_scan
                                              }.

It is enabled by default, but can be disabled using the -fno-simple-io command-line argument.

`interface`

When this feature is enabled, coqffi generates an inductive type which describes the set of primitives available, to be used with frameworks like FreeSpec or Interactions Trees .

Inductive CONSOLE : Type -> Type :=
| Echo : string -> CONSOLE unit
| Scan : unit -> CONSOLE string.

Definition inj_echo `{Inject CONSOLE m} (x0 : string) : m unit :=
  inject (Echo x0).

Definition inj_scan `{Inject CONSOLE m} (x0 : unit) : m string :=
  inject (Scan x0).

Instance Inject_MonadConsole `{Inject CONSOLE m} : MonadConsole m :=
  { echo := inj_echo
  ; scan := inj_scan
  }.

Providing an instance of the form forall i, Inject i M is enough for your monad M to be compatible with this featureSee for instance how FreeSpec implements it ). .

`freespec`

When this feature in enabled, coqffi generates a semantics for the inductive type generated by the interface feature.

Axiom unsafe_echo : string -> unit.
Axiom unsafe_scan : uint -> string.

Definition console_unsafe_semantics : semantics CONSOLE :=
  bootstrap (fun a e =>
    local match e in CONSOLE a return a with
          | Echo x0 => unsafe_echo x0
          | Scan x0 => unsafe_scan x0
          end).

Asynchronous Primitives

coqffi also reserves a special treatment for asynchronous primitives —i.e., functions which live inside the Lwt monad— when the lwt feature is enabled.

The treatment is very analogous to the one for impure primitives: (1) a typeclass is generated (with the _Async suffix), and (2) an instance for the Lwt monad is generated. Besides, an instance for the “synchronous” primitives is also generated for Lwt. If the interface feature is enabled, an interface datatype is generated, which means you can potentially use Coq to reason about your asynchronous programs (using FreeSpec and alike, although the interleaving of asynchronous programs in not yet supported in FreeSpec).

By default, the type of the Lwt monad is Lwt.t. You can override this setting using the --lwt-alias option. This can be useful when you are using an alias type in place of Lwt.t.

Exceptions

OCaml features an exception mechanism. Developers can define their own exceptions using the exception keyword, whose syntax is similar to the constructors’ definition. For instance,

exception Foo of int * bool

introduces a new exception Foo which takes two parameters of type int and bool. Foo (x, y) constructs of value of type exn.

For each new exception introduced in an OCaml module, coqffi generates (1) a so-called “proxy type,” and (2) conversion functions to and from this type.

Coming back to our example, the “proxy type” generates by coqffi is

Inductive FooExn : Type :=
| MakeFooExn (x0 : i63) (x1 : bool) : FooExn.

Then, coqffi generates conversion functions.

Axiom exn_of_foo : FooExn -> exn.
Axiom foo_of_exn : exn -> option FooExn.

Besides, coqffi also generates an instance for the Exn typeclass provided by the CoqFFI theory:

Instance FooExn_Exn : Exn FooExn :=
  { to_exn := exn_of_foo
  ; of_exn := foo_of_exn
  }.

Under the hood, exn is an extensible datatype , and how coqffi supports it will probably be generalized in future releases.

Finally, coqffi has a minimal support for functions which may raise exceptions. Since OCaml type system does not allow to identify such functions, they need to be annotated explicitly, using the =may_raise= annotation. In such a case, coqffi will change the return type of the function to use the =sum= Coq inductive type.

For instance,

val from_option : 'a option -> 'a [@@may_raise] [@@pure]

becomes

Axiom from_option : forall (a : Type), option a -> sum a exn.

Modules

Lastly, coqffi supports OCaml modules described within mli files, when they are specified as module T : sig ... end. For instance,

module T : sig
  type t

  val to_string : t -> string [@@pure]
end

becomes

Module T.
  Axiom t : Type.

  Axiom to_string : t -> string.
End T.

As of now, the following construction is unfortunately not supported, and will be ignored by coqffi:

module S = sig
  type t

  val to_string : t -> string [@@pure]
end

module T : S

Moving Forward

coqffi comes with a comprehensive man page. In addition, the interested reader can proceed to the next article of this series, which explains how coqffi can be used to easily implement an echo server in Coq.

Thomas Letan's Blog - ocaml

Peer-Programming in Modern OCaml with ChatGPT and Gemini

Peer-Programming in Modern OCaml with ChatGPT and Gemini

TL;DR

Editor Integration

Speech Recognition with Vosk

Prompting Local LLMs with Ollama

How It Started

More LLMs Lies

The Questionable Side Quest of Implementing a Connection Pool for cohttp-eio

Wrapping-up a Minimal Ollama Chat

Putting Everything Together

Final Words

Serving This Article from RAM for Fun and No Real Benefit

Serving This Article from RAM for Fun and No Real Benefit

Embedding My Website in a Binary

Serving the content with Dream

Naive Approach

Handling index.html Synonyms

Supporting Content-Type

Compressing if Requested

Caching

Building and Deploying the Website

Conclusion

On Vestigial Structures

On Vestigial Structures

Building Static Binaries for OCaml Projects

Building Static Binaries for OCaml Projects

Setting-Up a Dedicated dune Profile

Setting Up a Specific Opam Switch

Verifying the Resulting Binaries

Conclusion

Extensible Records in OCaml Using dmap

Extensible Records in OCaml Using dmap

Encoding Extensible Records

The Solution

Step 1: Forget About GADTs

Step 2: Deal with GADTs

Step 3: Profit

Neovim, OCaml Interfaces, Tree-Sitter and LSP

Neovim, OCaml Interfaces, Tree-Sitter and LSP

Writing a Function Whose Argument is a Polymorphic Function in OCaml

Writing a Function Whose Argument is a Polymorphic Function in OCaml

Implementing an Echo Server in Coq with coqffi.1.0.0

Implementing an Echo Server in Coq with coqffi.1.0.0

Project Layout

FFI Bindings

Sockets

Process Management

Implementing an Echo Server

Extracting and Building an Executable

Appendix

The Socket OCaml Module

The Proc OCaml Module

coqffi.1.0.0 In A Nutshell

coqffi.1.0.0 In A Nutshell

Getting Started

Requirements

Installing coqffi

Additional Dependencies

Primitive Types

Code Generation

Types

Pure values

Impure Primitives

simple-io

interface

freespec

Asynchronous Primitives

Exceptions

Modules

Moving Forward

The Questionable Side Quest of Implementing a Connection Pool for `cohttp-eio`

Handling `index.html` Synonyms

Supporting `Content-Type`

Setting-Up a Dedicated `dune` Profile

Extensible Records in OCaml Using `dmap`

Implementing an Echo Server in Coq with `coqffi.1.0.0`

The `Socket` OCaml Module

The `Proc` OCaml Module

`coqffi.1.0.0` In A Nutshell

Installing `coqffi`

`simple-io`

`interface`

`freespec`