Over the years I’ve come across more awesome Haskell tools, libraries and ideas than I can remember. I have put together Haskell notes, a place for me to keep notes about cool things I’ve found so in a year from now, when I run intop the problem solved by opne of those things, I’ll remeber it… I hope.

DRAFT

I have been thinking for quite some time about how a digital election system should work and the properties that one should hold. Now seemed like as good a time as any to write down these ideas and to see if others can fill in the gaps. A lot of the design I’ve been considering revolves around cryptography, for building both the identification infrastructure and for the ability to independently verify the results. If the properties below can be built with high confidence, then we could come to expect the same level of integrity in our elections that we expect from our banks …

I have been working on the assumption that the properties we expect from modern cryptography are similar to those we’d expect from a voting system, though there are further properties we’d also need to insure:

1. The design must be open to all and open to scrutiny
2. privacy should be guaranteed, only in the event that someone’s private key is compromised should it be possible to tell how someone voted (I can’t think of any way around this one, but it might be possible to provide electors with private keys they have no access to)
3. it should be possible for any elector to verify that their vote was submitted as they intended
4. it should be possible for any elector to verify that their vote was counted, and was counted how they expected
5. it should be possible for any elector to verify that the number of votes is equal to the number of votes submitted

I’m a fan of the idea of Liquid Democracy, so some other properties I would like to see may also be built on top of this base, but that’s probably for another post. The short version of what it means (at least to me) is all citizens can vote on all subjects, but may delegate this power at any time to another person for a particular subject, and may revoke that delegation at any time, creating a large incentive for people who choose to be representitives to vote the way they have promised to.

Open Protocols

Any advocate of cryptography will tell you that you shouldn’t trust any crypto which is protected by the fact its implementation is secret. Good crypto must stand up to the harsh light of public disclosure and scrutiny by experts. The same should be true for electronic voting; the protocol must be robust enough that any person can build their own client for casting a vote, and even be able to submit a valid vote created using their own implementation, without any fear that the particular implementation is up to no good. The same is true of the implementation of the server, anyone should be able to take the publicly released information for the authority running the election, and be able to reproduce the the result exactly.

• the implementation should not actually matter, anyone with sufficient skill should be able to build a client or server alternative and come to the same exact result as the official result

I’ve spent some time over the last week writing amazonka-s3-streaming () which provides a conduit based streaming interface for uploading files/data to Amazon’a S3 service. There’s also a way to upload files or strict ByteStrings to S3 concurrently using the async package. The package is based on Brendan Hay’s amazing Amazonka () suite of libraries.

I’ve tried to make things as efficient as possible, and to avoid excess memory usage if I can. A few of the ways I’ve attempted to do this are:

mmaping file segments

when concurrently uploading a file, each file part (currently 6MiB) is read in using mmap. The ensures that the mappings can be garbage collected as soon as they’re sent, usually resulting in much lower memory usage that mapping the whole file might.

The one big caveat here, which is unlikely to affect many people, but worth keeping in mind is that mmapping ByteStrings can break referential transparancy - if another process modifies the file, the contents of these parks might change.

Avoiding ByteString.Builder

Initially I was accumulating data in the conduit streaming interface using a ByteString Builder, and then converting this to a Lazy ByteString for upload. It occured to me that

1. the bytestrings are already in memory
2. by converting them to Builders, buffers would be allocated to produce the Lazy ByteString
3. these buffers would be immediately written to the socket
4. so why not just use the already allocated ByteStrings and send them to the socket.

so I changed the implementation to store a (D)List of ByteStrings until more than 6MiB have been accumulated, and this list is then turned into a conduit Source to be sent by the UploadPart request to S3. I also keep track of the length and the SHA256 hash and total length, which are needed by amazonka.

The Builder based implementation was my initial experiment, and didn’t survive long enough compare the performance with the list based implementation, so I don’t have benchmarks to prove this was a sensible move or not.

My experiments so far have shown that the concurrent upload is capable of saturating a 1Gbps connection quite easily, while using ~70MB RAM uploading a multi-hundred-megabyte file. The streaming interface has managed up to 40MB/s in my testing in GHCi. I’ll update this post once I’m back at work in the new year with some numbers.

I was inspired by a question by Alex Babkin about how to stream data into S3. His usecase is pretty cool, wanting to use his Qmuli package to stream data through AWS Lambdas written in Haskell without needing to download the entire file. Hopefully this package will allow this, while using no more than a few tens of MB more RAM than you would expect if you were able to stream the data directly over the network.

I’d love any feedback you’ve got, either here or on Reddit .

A few months ago, I was reading through Polarised Data Parallel Data Flow, and noticed there were some invariants which needed to be kept in mind when using the library, mainly that streams must only be consumed once. It seemed to me that with all the power of Haskell’s type system, we could do something about this.

I’d recently seen Gabriele Keller’s Bringing Down the Cost of Verification talk at LambdaJam ’16, and thought that maybe we could use the type system to ensure that

• Resources are only consumed once
• all resources are consumed

It turns out you can, but making it nice to work with it harder than I’d hoped. Let’s start with some imports, there’s a lot because we’re using some mildly advanced features in GHC.

{-# LANGUAGE KindSignatures #-}
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE TypeFamilies #-}
{-# LANGUAGE EmptyDataDecls #-}
{-# LANGUAGE MagicHash #-}
{-# LANGUAGE TupleSections #-}
{-# LANGUAGE ScopedTypeVariables #-}

module Main where

and some imports

import GHC.TypeLits
import GHC.Prim
import Data.Type.List
import Data.Proxy
import Control.Monad.Indexed

The most interesting import is Control.Monad.Indexed from the indexed package, which implements notion of indexed monads (and functors and comonads), which has been much more clearly explained elsewhere than I could have - thanks Conor!

The primary idea I wanted express in the type system was that of a counter which is incremented each time a resource is allocated, and a list of resources which are yet to be consumed. To achieve this, we’ll start with the L type, basically a type level tuple of a Nat (read type level Natural) and a list of Nats

data L (n :: Nat) (is :: [Nat])

The L type forms the input and output of our indexed monad. The input Nat tells you how many says how many resources have been allocated before a particular action was invoked, and the output Nat minus the input tells you how many resources were allocated within an action. Similarly, the input and output [Nat]’s tell you which resources haven’t been consumed before and after a particular action. This will make more sense once we’ve introduced the Linear monad.

Linear is essentially a monad transformer defined by a newtype wrapping a monad p, with some extra type parameters

newtype Linear p i o a = Linear (p a)

Now for the indexed monad classes we’ll need:

-- ireturn is similar to return or pure, but guarantees
-- through its type that no effects which could change
-- the indices occur.
class IxFunctor m => IxPointed m where
  ireturn :: a -> m i i a

-- imap can only change the last type parameter, just like our
-- old friend fmap. Anything that changed in the indexed
-- parameters will still change after being imapped
class IxFunctor f where
	imap :: (a -> b) -> f j k a -> f j k b

-- iap is where things get more interesting - it works
-- just like ap or <*>, but also composes the effects
-- expressed in the indices.
--
-- Notice that the final indices are i and k, and the
-- two actions share j.
class IxPointed m => IxApplicative m where
	iap :: m i j (a -> b) -> m j k a -> m i k b

-- ibind is just like >>= but again composing the indices
-- as above.
class IxApplicative m => IxMonad m where
	ibind :: (a -> m j k b) -> m i j a -> m i k b

The implementations for Linear are trivial, just using non-indexed functions we’re used to in everyday Haskell, with the necessary changes in IxMonad to work with the newtype.

instance Functor p => IxFunctor (Linear p) where
  imap f (Linear x) = Linear (fmap f x)

instance Applicative p => IxPointed (Linear p) where
  ireturn x = Linear (pure x)

instance Applicative p => IxApplicative (Linear p) where
  iap (Linear pf) (Linear px) = Linear (pf <*> px)

instance Monad p => IxMonad (Linear p) where
  ibind f (Linear a) = Linear (a >>= \x -> let (Linear y) = f x in y)

With this sorted, we can get into the details of managing resources. First, we’ll define a newtype which wraps a Resource, and whose type includes the index it was assigned when it was allocated.

newtype Resource (n :: Nat) a = Res a

So how do we allocate a Resource? with allocate

allocate :: (m ~ (n + 1)          -- (3)
            , os ~ (Insert n is)  -- (4)
            , Functor p)
         => p a                   -- (1)
         -> Linear p
                  (L n is)        -- (2)
                  (L m os)        -- (5)
                  (Resource n a)  -- (6)
allocate x = Linear $ fmap Res x

There’s a lot going on in this type, so let’s walk through it.

1. The action from the underlying monad is is used to allocate the resoure is passed in. this could be, for example openFile which allocate a file Handle
2. the Linear type is passed n, the current resource allocation count and is, the in scope resources.
3. Since we’re allocating a new resource, we need to increment the outgoing count
4. the output has the current index n inserted into the inout - we’ve consumed nothing which was previously in scope, and now we’ve got one more, n
5. the previous two values are used in the output, to be passed to any following actions.
6. the wrapped resource is returned, marhed with its index.

The other side of this is resource consumption. To consume a resource, we need to ensure that the given resource hasn’t been consumed elsewhere, and that once it is consumed, we remove its index from the list of in scope resources.

consume :: (Find i is ~ 'True, os ~ Remove i is)
        => Resource i a
        -> (a -> p b)
        -> Linear p (L n is) (L n os) b
consume (Res x) f = Linear (f x)

Here we first check that index i is present in the input list of resources is. We also remove set the output list os to the input list with i Removed. Then we pass in the wrapped resource and the function in the underlying monad which will deallocate the resource.

Along similar lines, we could implement a function for using the resource while it’s in scope but which doesn’t deallocate it - if you wanted to ensure that resources were only ever used once, as in the original problem, then we would omit this function.

utilise :: (Find i is ~ 'True)
        => Resource i a
        -> (a -> p b)
        -> Linear p (L n is) (L n is) b
utilise (Res x) f = Linear (f x)

Note that the only change is that i is not removed from the output, and everything else is identical to consume.

Finally we can provide a function to run a computation, which ensures that the resource index starts at zero, and ensures that all resources are consumed.

runLinear :: KnownNat m => Linear p (L 0 '[]) (L m '[]) a -> p a
runLinear (Linear x) = x

which can be used like so

main :: IO ()
main = runLinear $
  allocate openFile >>>= \f0 ->
  allocate openFile >>>= \f1 ->
  consume f0 closeFile >>>= \_ ->
  allocate openFile >>>= \f2 ->
  consume f2 closeFile >>>= \_ ->
  consume f1 closeFile

Notice that the resources are consumed in a different order from the order they were allocated. This is something that differentiates this technique from something like Golang’s defer.

Now for the problem. Using the library works well when used like this, but it doesn’t allow you to define composable actions. The problem is that GHC isn’t s enough to smart enough when composing actions to realise that Find i (Insert i is) is always true when it doesn’t know what is is. I had tried just cons’ing i onto is but this only helps in the situation where a resource being consumed is the most recently allocated one. For example, I can define test1 with the type shown, but I can’t define it with either of the other two, which are the types GHC tried to infer

-- test1 :: Linear IO (L n '[]) (L (n+3) '[]) ()
-- test1 :: Linear IO (L n '[]) (L (((n+1)+1)+1) '[]) ()
test1 :: Linear IO (L 0 '[]) (L 3 '[]) ()
test1 =
  allocate openFile >>>= \f0 ->
  allocate openFile >>>= \f1 ->
  consume f0 closeFile >>>= \_ ->
  allocate openFile >>>= \f2 ->
  consume f2 closeFile >>>= \_ ->
  consume f1 closeFile

To solve this, I’d love to see type level Sets which allow for tests like Find i (Add i s) to be true for all s.

I’d love to hear what others have to say about this idea, and alternative methods which can be used to implement similar ideas.