Pencil Code Project Ideas

Microdatabase Project Idea

This is a project plan that could be followed to create a microdatabase that would be interesting for beginners.

Basic Design

The client API would have three parts:

  1. A constructor for an object associating the database with a json file on the server. Doesn't do much, just remembers the URL.
    mydb = new JsonDb('mydb.json')
    
  2. A simple key-value store that saves its values textually in the JSON file. (The methods should return Promises, which in the future will work with an await an extension to CoffeeScript. See the await syntax project.)
    mydb.get('somefield').then (x) -> alert(x)
    val = await mydb.get 'somefield'
    await mydb.set 'somefield', 'somenewvalue'
    

    This would be implemented via a JSON GET/POST request that transmit JSON values and responses like this:

    GET /data/mydb.json?get=somefield
    GET /data/mydb.json?set=somefield&value="somenewvalue"
    
  3. A transactional interface that allows complex functions to be evaluated on the server.
    matchTransaction = mydb.transaction (x, y) ->
      result = []
      for record in this.records
        if record.x is x and record.y >= y
          result.push record
      result.sort (a, b) -> a.y - b.y
      return result
    results = await matchTransaction x, y
    

    This would be implemented via another JSON request that transmits a transaction as a Javascript string and returns a JSON response:

    get /data/mydb.json?arg0="foo"&arg1=15&transaction=function(x, y){ var result=[]; for (var _i = 0; _i<this.records.length; ++_i) { var record = this.records[i]; if (record.x === x && record.y >= y) { result.push(record); } } result.sort....
    
    The Javascript string would just be a function definition written in standard Javascript, the same thing you get when you use fn.toString() on a function in Javascript.

Phase 1: build a nodejs server for get/set

In this phase, we would build a basic node.js server that uses Express and listens to the /data URL, implementing the get and set API.

When setting data for the first time, The request file should be created as a UTF-8 text file and initialized with an empty JSON object document '{}'. Properties should be able to be added and queried via the server.

Phase 2: implement a client library and multiple users

In this phase, we would implement multiple users, using the convention that the username is the first component of the domain name in the Host header, and when you request "http://david.pencilcode.net/data/myfile.json", it saves the data in "david/myfile.json".

And we would also implement a client library and create some tests.

Phase 3: link a javascript-in-javascript interpreter.

To safely execute Javascript transactions, we must prevent them from running too long or doing dangerous things. So we want to interpret transactions inside a Javascript interpreter within Javascript.

So the second phase is to embed an interprter like this one, and use it to implement transactions.

In this phase, we would add tests to verify that basic things can be done with the interpreter, but that dangerous things, such as accessing node.js globals, cannot be done.

Phase 4: implement resource limits

To make it safe to provide the database for free to all students, it must automatically limit each database to a small amount of storage (say, 1mb), and a small amount of CPU per transaction.

Exceeding these limits should return an clear error.

In this phase, we would add tests of the resource limits.

Phase 5: implement an in-memory cache

If several queries or transactions come in for the same file, the file shouldn't be re-loaded from disk each time. Instead, it should be retained in RAM for a few minutes.

After a short time without access, databases should not be kept in RAM.

Phase 6: implement passwords and "stored procedures"

In this phase, we would add a couple methods to allow a database to store specific allowable transactions, and limit access so that users need to provide a "secret password" to use a particular transaction.